Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion

Welcome to Day 5 of Week of Continuous Intrusion tools. We are discussing security of Continuous Integration (CI) tools in this series of blog posts.

Day 1 – Jenkins (and Hudson) (Click Here)
Day 2 – TeamCity (Click Here)
Day 3 – Go and CruiseControl (Click Here)
Day 4 – Common Abuse Set, Lateral Movement and Post Exploitation (Click Here)
Day 5 – Defense and other discussion

Day 5, the last day of the Week of Continuous Intrusion Tools, is dedicated to defenses against the attacks we discussed both for those who develop these tools and users, responses to those who reported “things”, instances of companies getting 0wned using CI tools and previous work.

Lets get started with Defense.

Defense for Administrators/Users

  • No build agent/executor should run on the master EVER. If the CI tool you use installs it by default (see the Common Abuse Set in Day 4), keep a strict control over which project or build can use the agent on master. If there are certain cases where you require agent on maser, try to eliminate those requirements by all means. 
  • Ability to Configure builds should be very restricted. Ask yourself, if you will you provide Local Administrator access to all the executors/agents machines tied to a project to a user.
  • Make sure that agents are pooled for projects. No project should be allowed to use arbitrary agents.
  • Secure the Administrator web console/dashboard.
    •  Make sure to enable Authentication and use complex passwords (at least) for Admin users, even if the tool runs only internally.
    • Enable HTTPS.
    • Consider integration with Active Directory for authentication. Please make sure to weigh pros and cons specially if your CI tool is exposed to the internet.
  • Ask your users to not use their username as passwords. 
  • Try not to expose your CI Tool to the Internet (unless you want to show to the World how your developers leave sensitive data in build logs). VPN anyone? (You may also like to experiment with things similar to Jenkins’ Google or OpenID login plugins)
  • Do NOT provide even read access to Anonymous users (unless, once again, you want to show to the World how your developers leave sensitive data in build logs).
  • Ask your Red Team or Penetration Testers to specifically target your CI tool.

Defense for Tool Developers

    • Enforce password policies (complexity, expiry, captcha on login failures etc.)
    • A user must install agent/slave explicitly on the master. Do not include it in the install package. Show big red warning regularly if an agent/slave is installed on the master.
    • Please, puppy please, do not store credentials and keys in clear text on master (or anywhere else).
    • Assume that your users don’t know a thing about security. Make your tool at least reasonably secure in the default installation. IMHO, because “default just works”, the level of security in a default installation is what one finds in majority of installations. You can learn from each other, for example:
      • When TeamCity can enable authentication in the default installation, why can’t Jenkins and Go? 
      • When Go does not install an agent on master in the default install, why can’t Jenkins and TeamCity? and so on.
    • Explore the possibility of running as a non-SYSTEM or admin on Windows installations.

    Response to those who reported things (and those who didn’t)

    Response of CI tools developers to the above things has been, in the most polite words, unsatisfactory. Lets have a look at some examples. Warning: May look like a rant!
    In August 2014, I blogged about how Jenkins could be abused. While I did not report anything (as there was no vulnerability and I do not believe in responsible disclosure – more on it later on), someone raised an issue in the Jenkins issue tracker system. https://issues.jenkins-ci.org/browse/JENKINS-24513
    Some gems from that issue:

    • Response: “this is just something to be documented” Status: Not done yet.
    • Response: “How is this NOT obvious?”  Because it is not well documented.

    Those who tried to report issues: http://www.th3r3p0.com/vulns/jenkins/jenkinsVuln.html

    Couple of gems from the above:
    • “acknowledge that the Jenkins admins have the power to access all of these credentials and don’t make unstrusted people admins”
    • “But admins are admins”
    Someone posted link to Day 3 on Go Google Group: https://groups.google.com/forum/#!topic/go-cd/TK9hzjO_CsA
    Couple of interesting responses:
    • “For obvious reasons, that’s a bad idea” – Exactly my point.
    • “for it to be considered a pen test, I’d have expected it to be more thorough and trying different attacks” – The Trivial Attack Syndrome.
    After my talk at BlackHat Europe, saw this comment on TeamCity by a developer (don’t want to take names): “as usually security goes against usability, so default setup should not be considered secure.” Sad that this comes from those who, like everyone else, “take security very seriously”.

     

    Free Advice: Dealing with security issues reported/not reported.

    Dear $Vendor, I know its very hard to listen to someone who tells you what is wrong with your software. Specially, if it is some random person from the Internet who may not know a bit of what you know about your code or tool. I understand that there are more serious issues and problems to be fixed. Still, try to follow at least the below points:
    • “If its admin its game over” is so 90s. Move on. Try to minimize lateral propagation and post exploitation.
    • Once again, the default security is the security you will find in majority of installs.
    • Don’t try to be dismissive of the reports. If you will be dismissive, potential issue raisers will just move on. 
    • FFS, improve documentation. It is not enough to mark issues as “obvious” in comments. Make it look obvious in the documentation. 
    • All reported/not reported issues will not be 0 days. There will be issues which are critical enough but do not provide unauthenticated remote code execution. Understand that various low and medium severity issues are chained together for compromise. 
    • Repeat with me, trivial attacks are more dangerous, trivial attacks are more dangerous, trivial attacks…
    I would like to make a note here that couple developers of Go and TeamCity reached out to me in October 2015 when I was tweeting random stuff about their tools. I shared the relevant parts of my BlackHat and DeepSec preso with them and we had a good (but no fruitful) discussion.

     

    Why I don’t do “Responsible Disclosure”

    Because I have no time, energy and motivation to do so. I sit on my bugs and use them during my pen tests and Red Team engagements or occasionally in the trainings I do. Also, bug bounty sucks!

    Getting 0wned due to CI Tools

    Previous Work

    Is this your pipe? Hijacking the build pipeline. (Defcon 22): https://goo.gl/9iTHmz

     

    What we didn’t discuss?

    • Masteragent communication and MITM attacks.->
    • Compromising integrity of the source code and the build process.
    • Web application related vulnerabilities of web consoles.
    • Memory corruption bugs.  
    • Logging process and issues.
    I would like to end the week with some words of wisdom – As a single withered tree, if set aflame, causes a whole forest to burn, so does a single improperly configured Continuous Integration tool destroys a whole enterprise network. – Chanakya (350-275 BCE)
    Hope you enjoyed the week as much as I enjoyed creating it. Please leave comments and feedback.

    To support my research, join me for a two days training “PowerShell for Penetration Testers” at:

    BlackHat, Asia (March 29-30th, 2016) – https://www.blackhat.com/asia-16/training/powershell-for-penetration-testers.html

    HITB, Amsterdam (May 24-25th, 2016) – http://conference.hitb.org/hitbsecconf2016ams/sessions/2-day-training-3-powershell-for-penetration-testers/

    Welcome to Day 5 of Week of Continuous Intrusion tools. We are discussing security of Continuous Integration (CI) tools in this series of blog posts.

    Day 1 – Jenkins (and Hudson) (Click Here)
    Day 2 – TeamCity (Click Here)
    Day 3 – Go and CruiseControl (Click Here)
    Day 4 – Common Abuse Set, Lateral Movement and Post Exploitation (Click Here)
    Day 5 – Defense and other discussion

    Day 5, the last day of the Week of Continuous Intrusion Tools, is dedicated to defenses against the attacks we discussed both for those who develop these tools and users, responses to those who reported “things”, instances of companies getting 0wned using CI tools and previous work.

    Lets get started with Defense.

    Defense for Administrators/Users

    • No build agent/executor should run on the master EVER. If the CI tool you use installs it by default (see the Common Abuse Set in Day 4), keep a strict control over which project or build can use the agent on master. If there are certain cases where you require agent on maser, try to eliminate those requirements by all means. 
    • Ability to Configure builds should be very restricted. Ask yourself, if you will you provide Local Administrator access to all the executors/agents machines tied to a project to a user.
    • Make sure that agents are pooled for projects. No project should be allowed to use arbitrary agents.
    • Secure the Administrator web console/dashboard.
      •  Make sure to enable Authentication and use complex passwords (at least) for Admin users, even if the tool runs only internally.
      • Enable HTTPS.
      • Consider integration with Active Directory for authentication. Please make sure to weigh pros and cons specially if your CI tool is exposed to the internet.
    • Ask your users to not use their username as passwords. 
    • Try not to expose your CI Tool to the Internet (unless you want to show to the World how your developers leave sensitive data in build logs). VPN anyone? (You may also like to experiment with things similar to Jenkins’ Google or OpenID login plugins)
    • Do NOT provide even read access to Anonymous users (unless, once again, you want to show to the World how your developers leave sensitive data in build logs).
    • Ask your Red Team or Penetration Testers to specifically target your CI tool.

    Defense for Tool Developers

      • Enforce password policies (complexity, expiry, captcha on login failures etc.)
      • A user must install agent/slave explicitly on the master. Do not include it in the install package. Show big red warning regularly if an agent/slave is installed on the master.
      • Please, puppy please, do not store credentials and keys in clear text on master (or anywhere else).
      • Assume that your users don’t know a thing about security. Make your tool at least reasonably secure in the default installation. IMHO, because “default just works”, the level of security in a default installation is what one finds in majority of installations. You can learn from each other, for example:
        • When TeamCity can enable authentication in the default installation, why can’t Jenkins and Go? 
        • When Go does not install an agent on master in the default install, why can’t Jenkins and TeamCity? and so on.
      • Explore the possibility of running as a non-SYSTEM or admin on Windows installations.

      Response to those who reported things (and those who didn’t)

      Response of CI tools developers to the above things has been, in the most polite words, unsatisfactory. Lets have a look at some examples. Warning: May look like a rant!
      In August 2014, I blogged about how Jenkins could be abused. While I did not report anything (as there was no vulnerability and I do not believe in responsible disclosure – more on it later on), someone raised an issue in the Jenkins issue tracker system. https://issues.jenkins-ci.org/browse/JENKINS-24513
      Some gems from that issue:

      • Response: “this is just something to be documented” Status: Not done yet.
      • Response: “How is this NOT obvious?”  Because it is not well documented.

      Those who tried to report issues: http://www.th3r3p0.com/vulns/jenkins/jenkinsVuln.html

      Couple of gems from the above:
      • “acknowledge that the Jenkins admins have the power to access all of these credentials and don’t make unstrusted people admins”
      • “But admins are admins”
      Someone posted link to Day 3 on Go Google Group: https://groups.google.com/forum/#!topic/go-cd/TK9hzjO_CsA
      Couple of interesting responses:
      • “For obvious reasons, that’s a bad idea” – Exactly my point.
      • “for it to be considered a pen test, I’d have expected it to be more thorough and trying different attacks” – The Trivial Attack Syndrome.
      After my talk at BlackHat Europe, saw this comment on TeamCity by a developer (don’t want to take names): “as usually security goes against usability, so default setup should not be considered secure.” Sad that this comes from those who, like everyone else, “take security very seriously”.

       

      Free Advice: Dealing with security issues reported/not reported.

      Dear $Vendor, I know its very hard to listen to someone who tells you what is wrong with your software. Specially, if it is some random person from the Internet who may not know a bit of what you know about your code or tool. I understand that there are more serious issues and problems to be fixed. Still, try to follow at least the below points:
      • “If its admin its game over” is so 90s. Move on. Try to minimize lateral propagation and post exploitation.
      • Once again, the default security is the security you will find in majority of installs.
      • Don’t try to be dismissive of the reports. If you will be dismissive, potential issue raisers will just move on. 
      • FFS, improve documentation. It is not enough to mark issues as “obvious” in comments. Make it look obvious in the documentation. 
      • All reported/not reported issues will not be 0 days. There will be issues which are critical enough but do not provide unauthenticated remote code execution. Understand that various low and medium severity issues are chained together for compromise. 
      • Repeat with me, trivial attacks are more dangerous, trivial attacks are more dangerous, trivial attacks…
      I would like to make a note here that couple developers of Go and TeamCity reached out to me in October 2015 when I was tweeting random stuff about their tools. I shared the relevant parts of my BlackHat and DeepSec preso with them and we had a good (but no fruitful) discussion.

       

      Why I don’t do “Responsible Disclosure”

      Because I have no time, energy and motivation to do so. I sit on my bugs and use them during my pen tests and Red Team engagements or occasionally in the trainings I do. Also, bug bounty sucks!

      Getting 0wned due to CI Tools

      Previous Work

      Is this your pipe? Hijacking the build pipeline. (Defcon 22): https://goo.gl/9iTHmz

       

      What we didn’t discuss?

      • Master<->agent communication and MITM attacks.
      • Compromising integrity of the source code and the build process.
      • Web application related vulnerabilities of web consoles.
      • Memory corruption bugs.  
      • Logging process and issues.
      I would like to end the week with some words of wisdom – As a single withered tree, if set aflame, causes a whole forest to burn, so does a single improperly configured Continuous Integration tool destroys a whole enterprise network. – Chanakya (350-275 BCE)
      Hope you enjoyed the week as much as I enjoyed creating it. Please leave comments and feedback.

      To support my research, join me for a two days training “PowerShell for Penetration Testers” at:

      BlackHat, Asia (March 29-30th, 2016) – https://www.blackhat.com/asia-16/training/powershell-for-penetration-testers.html

      HITB, Amsterdam (May 24-25th, 2016) – http://conference.hitb.org/hitbsecconf2016ams/sessions/2-day-training-3-powershell-for-penetration-testers/


      Print Share Comment Cite Upload Translate
      APA
      () » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion. Retrieved from https://www.truth.cx/2015/12/04/week-of-continuous-intrusion-tools-day-5-defense-and-other-discussion/.
      MLA
      " » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion." - , https://www.truth.cx/2015/12/04/week-of-continuous-intrusion-tools-day-5-defense-and-other-discussion/
      HARVARD
      » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion., viewed ,
      VANCOUVER
      - » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion. [Internet]. [Accessed ]. Available from: https://www.truth.cx/2015/12/04/week-of-continuous-intrusion-tools-day-5-defense-and-other-discussion/
      CHICAGO
      " » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion." - Accessed . https://www.truth.cx/2015/12/04/week-of-continuous-intrusion-tools-day-5-defense-and-other-discussion/
      IEEE
      " » Week of Continuous Intrusion Tools – Day 5 – Defense and other discussion." [Online]. Available: https://www.truth.cx/2015/12/04/week-of-continuous-intrusion-tools-day-5-defense-and-other-discussion/. [Accessed: ]
      Select a language:




      Related Posts