Posts
U.S. Gov Employee Responds to TrustedSec's Review of Healthcare.gov
After I wrote yesterday's article "The Questionable Value and Ethics of TrustedSec's Pen Test of the HealthCare.gov Website", I received an email from a well-respected employee of a large government agency who had read TrustedSec's report on the Healthcare.gov website. This employee has asked me if I would publish the content of that email on my blog. Here it is with some minor formatting changes.
-------------------
So let's put aside the isc2 ethics violation by TrustedSec that this "report" is and instead focus upon its content."
The report is split into two parts, one based upon public open source intel gathering, and on upon actual "analysis". Contrary to what Goebbels might say, repeating a lie does not make it true. The first half of the "analysis" consists of misquotes and out of context statements about news reports, blog postings and the Heritage foundation (an anti-Affordable Care Act org).
This lack of substance extends to the second part of the "analysis" which shows a lack of understanding of both what healthcare.gov is and what security is.
They raise issues with things like the Terms of Service (TOS).
They raise issues with data.healthcare.gov.
Healthcare.gov is not just a website, it is a complex node in a web of Federal, State, and private systems that interconnect to produce the healthcare.gov site. The data in it comes from state exchanges, medicare, the IRS, SSA, and other Federal/state agencies, plus private insurers. It's not just a webserver/webapp with a back end database like something circa 2003.
These two examples show the lack of due care conducted on this analysis. Please take a moment to read the "results" [CARR: A link to TrustedSec's report is provided below]. The level of writing and actual deliverable are so laughable that if a contractor had produced this for my agency I would have terminated their contract on the spot. (The report shows) no due diligence, sloppy work, and worst of all it is wrong in its "conclusions".
------------------------
Here's a link to TrustedSec's public report (.pdf) for those readers who wish to review it and assess the above criticism for themselves. Comments are open.
UPDATE (12/13/13): "On December 11, in order to address ongoing questions, Committee members and staff received a classified briefing from Dr. Kevin Charest, the HHS Chief Information Security
Officer, and Ned Holland, HHS Assistant Secretary for Administration. Portions of this briefing
were classified to protect information relevant to national security. This memo contains a summary of the unclassified portion of the briefing."
-------------------
So let's put aside the isc2 ethics violation by TrustedSec that this "report" is and instead focus upon its content."
The report is split into two parts, one based upon public open source intel gathering, and on upon actual "analysis". Contrary to what Goebbels might say, repeating a lie does not make it true. The first half of the "analysis" consists of misquotes and out of context statements about news reports, blog postings and the Heritage foundation (an anti-Affordable Care Act org).
They extrapolate from news articles and jump to conclusions that would be laughed out of a Bsides conference, let alone a court of law. Most of the "observations" are generic in nature with no supporting detail. Everything is anecdotal. Everything is hearsay. There is no direct observation of any vulnerability, and only "potential risks".
Many of the articles highlight pre-launch issues that have since been resolved, and others are issues common to most web application (hello, user enumeration? Seriously? Any site with a unique user account has this issue).
This lack of substance extends to the second part of the "analysis" which shows a lack of understanding of both what healthcare.gov is and what security is.
In the professional world of cyber security there are two concept at the heart of computer forensics; peer review and reproducibility. Professionals understand that their word is not enough and they actually have to show something that the community and their peers can reproduce. None of their findings are "reproducible" vulnerabilities. They are all vague possible-maybe-there-could-be risks, or worse yet, a gross misunderstanding of what they are "analyzing."
They raise issues with things like the Terms of Service (TOS).
They raise issues with data.healthcare.gov.
Healthcare.gov is not just a website, it is a complex node in a web of Federal, State, and private systems that interconnect to produce the healthcare.gov site. The data in it comes from state exchanges, medicare, the IRS, SSA, and other Federal/state agencies, plus private insurers. It's not just a webserver/webapp with a back end database like something circa 2003.
They raise an issue that data will be shared with outside agencies which shows they don't understand what healthcare.gov is. Then they raise another issue about public profiles on the data.healthcare.gov site. The fact is that Data.healthcare.gov is an open data initiative based on the data gathered from insurers. Public profiles are a feature, not a bug, of that SEPARATE platform.
These two examples show the lack of due care conducted on this analysis. Please take a moment to read the "results" [CARR: A link to TrustedSec's report is provided below]. The level of writing and actual deliverable are so laughable that if a contractor had produced this for my agency I would have terminated their contract on the spot. (The report shows) no due diligence, sloppy work, and worst of all it is wrong in its "conclusions".
Determinations need proof beyond media quotes and theoretical issues. They need to be based in fact.
------------------------
Here's a link to TrustedSec's public report (.pdf) for those readers who wish to review it and assess the above criticism for themselves. Comments are open.
UPDATE (12/13/13): "On December 11, in order to address ongoing questions, Committee members and staff received a classified briefing from Dr. Kevin Charest, the HHS Chief Information Security
Officer, and Ned Holland, HHS Assistant Secretary for Administration. Portions of this briefing
were classified to protect information relevant to national security. This memo contains a summary of the unclassified portion of the briefing."
Haha - always entertaining to say the least. Wish they would use their actual names instead of hiding behind cloak and daggers. Easy to say everything is peachy on the other side behind mirrors. Here's a thought - give us permission to do a test, free of charge and let us publish the results. I can only see systemic issues on how this was developed. On the bsides comments: Haven spoken at over 30 BSIDES and put on the conference - I would disagree with that statement as I am sure would the community.
ReplyDeleteUnfortunately this post is based on emotion - another example of inaccuracy and biased. The truth is people that are on the inside that are actively working on this know its a trainwreck and have expressed so in conversations with me - that security was an afterthought just as we can show through the open source analysis. The truth is you are to close to this to give a position that isn't based on ridiculous accusations and insults. Troll on my friend.
At least I got a good laugh out of this - made my day =)
ReplyDeleteOne last thing before I dismiss this completely - instead of having a discussion about these issues, you attack. Security has no political boundaries, the areas identified are exposures and continue to be exposures. Almost every single exposure identified in the report and through others were done post launch, I can confirm each and every one of them for you if you would like and the dates post launch. What this shows unfortunately is people trying to fight for their jobs for a major screw up on the development process, and showing up outside of HHS in anger and fear. It's an unfortunate situation, but one that wasn't caused by us.
ReplyDeleteThis post just confirmed it for us - we as Americans need to be extremely cautious of the website, its people that are working on it, and the ability to protect our information. Thank you.
This doesn't make sense and reads like the Unconfirmed Employee did not read the account or did not understand the vulnerabilities presented.
ReplyDeleteBased on Dave's team's research, the datestamp of the blogs included post-production of the .gov site, and the numerous mistakes, there was a classification made of the website that it likely has issues.
Note that the Unconfirmed Employee does not take the time to dismiss each security concern, but glosses over specific (and documented) vulnerabilities to mention the "low" findings. On their own, I would say these low findings were unconcerning, but so many combined usually indicates a lack of understanding. This is anecdotal evidence, but usually these cumulative bad decisions are evidence of developers being asleep at the wheel.
It's my opinion that the above email was written by someone who failed to understand the security vulnerabilities and the evidence provided, and I would like them to speak "On the record" in order to rebut my concerns about the source.
Very valid points Byran and well said. I would like to address the unconfirmed employees comments here in a respectful manner versus what was done to me as being called a Nazi and someone who would be laughed out of BSIDES.
ReplyDelete1. When were we discussing computer forensics? I'm confused - the issues identified were a cause of lack of secure coding practices and appropriate controls in place to protect the data.
2. On the reproducible aspect - all of them are reproducible, I'm extremely confused on this. I've shared the details with individuals working on the site to fix it. I would be happy to do the exact same thing with you. What has already been fixed is reproducible per the report, the others were not as they have not been fixed and expose critical information including the ability to put malicious content on healthcare.gov which can be executed by anyone visiting the site. Do you need the reproducible content? I am more than happy to provide.
2. On the complex nature of healthcare.gov - I would assume that is highly complex in nature - the integration it has into many of the different agencies and state exchanges is a complex one - completely independent systems, technologies, etc. This makes security even more of an importance on what should have been on the roadmap. The "data hub" does integration into several places - this is a concern - what security is around that?
3. On the data.healthcare.gov issue - are you stating that the website which posts information on healthcare.gov and supports the infrastructure for pulling information is irrelevant to the overall security of the architecture and has no significant impact on the potential compromise of healthcare.gov? I would challenge this and claim it false as there are so many different pieces of integration into healthcare.gov. For example, chat.healthcare.gov - if that was compromised and individuals seeking help were hacked each time they clicked it and their information exposed - would that not be just as detrimental as an exposure on healthcare.gov as it's all in the same supporting infrastructure? You mention two "SEPARATE" systems in bold. Why is a full reverse proxy implemented and launched off of healthcare.gov natively to provide its content on healthcare.gov if they are two "SEPARATE" systems? So there is no integration between the two? I would argue this fact and challenge it.
4. On the topic of these issues were pre production - I can produce screenshots, messages, and others on how this is inaccurate and not factually accurate. Let me know where to send the emails to on how to reproduce and the timestamps and post production issues.
I'll disregard the emotional statements again around the report writing as based on your Nazi and BSIDES comments completely disregard your opinion in this manner and further aid what we all need to be concerned with. More than happy to have a discussion about this and any topics you have - unfortunately your information isn't accurate and full of emotional responses that completely set the opposite tone of what I would have hoped you wanted.
Instead of arguing in all of this, our goal should be to figure out what happened, why this is such a mess and how do we prevent this in the future. At the end of the day security draws no lines politically - it focuses on making things better and bringing awareness to minimize the risk to the organization and the information. This wasn't done here and its apparent. Say what you will, argue what you will - and I hope I'm wrong - security was an afterthought of this and its unfortunate that politics has played in the diversion techniques we see here to actually address the concerns, instead push blame somewhere else.
I hope something *better* comes of all of this which has always been my intention. Bring security awareness information to the people that can make the difference.
Dave, I've just finished reading the Public Information Analysis portion of your report. In my opinion, it hurt your overall focus because the sources were either very poor or the findings were outdated (meaning fixed). I'm not sure why you or your team thought it would be a good idea to include it unless it was something that the Committee asked you to do (i.e., a survey of what's been published). Even if that was the case, you compiled third and fourth person accounts from sometimes highly questionable sources.
ReplyDeleteThat was the whole intent was to publish data that had already been fixed =) in the sources you mention as sketchy, the findings were reproducible or had sources that were accurately fact checked. Regardless of bias, they had information that backed the claims on the security front. I would be happy to review which ones you have questions on but the analysis we performed showed significant strength on what was being discussed or the findings were directly validated.
ReplyDeleteAnd the intent was to show other sources of findings, not just solely coming from us as being the one stop shop for everything being found - there's a lot of others that have came to the same conclusion or found very similar exposures. Again - based on responsible disclosure - we didn't want to publish anything that hadn't been fixed directly yet.
ReplyDeleteHere are some of the worst - referenced by number:
ReplyDelete2.1 is meaningless since pretty much all gov sites are attacked every day, as well as major corp sites for that matter.
2.4 was not only quickly fixed but the researcher who reported it complimented the Healthcare.gov team for being so responsive.
2.6 is the Heritage Foundation who's "cyber security expert" is a retired Army officer with zero security engineering credentials.
2.7 The Daily Caller, a Right-wing publication discussed an ineffective hacking tool.
2.7 Popular Mechanics and APNEWS and Forbes articles were all re-hashes of prior mentioned reports.
2.7 The FedScoop article interviewed some cyber security experts but none disclosed any vulnerabilities - they merely expressed their opinions.
2.1 - the whole point there is missed - wasn't to say that sites aren't attacked - the fact that they only saw 16 after months of launch is highly inaccurate - this one was a major foundation for the lack of detection capabilities
ReplyDelete2.4 Nothing wrong with getting compliments for fixing the issue - fixing the issues are a great thing.
2.6 Sorry - I have to knock you on this one, screenshots were provided, the individual affected was notified and it had visibility in all levels of government. Just because they didn't interview someone you liked doesn't hinder the fact that there was a serious glitch that exposed someone else's information.
2.7 - Nothing wrong with re-hashed reports if they get the accuracies of what occurred.
On the "right wing" website, you are missing the point of all of these reports, they are reported by multiple dozens. The "hacker" tool actually made the majority of them. These are different sources of information. I really can't go back and forth with this when your entire intent is to discredit and disassemble each part. The sources and information check out and so do the facts. Nothing you have mentioned show anything different.
This comment has been removed by the author.
ReplyDeleteHad the same links posted twice - heres the corrected post:
DeleteHere's like 6 other sites saying the same thing, are Christians evil too?
http://www.csmonitor.com/Innovation/2013/1113/Hacking-tool-threatens-Healthcare.gov-site
http://www.ubergizmo.com/2013/11/destroy-obamacare-hacking-tool-discovered-to-target-healthcare-gov/
http://www.bloomberg.com/video/the-new-hacking-tool-destroy-obamacare-PDBJSXi9Tk6Of68QLrkUJg.html
http://www.informationweek.com/security/vulnerabilities-and-threats/hackers-threaten-destruction-of-obamacare-website/d/d-id/1112207?
http://www.addictinginfo.org/2013/11/20/destroy-obamacare-cyber-attackers/
http://www.businessweek.com/videos/2013-11-14/the-new-hacking-tool-destroy-obamacare
The whole purpose of that one specifically was to show that people are actively going after the site - that it's not a hypothetical, evidence points to targeted attacks.
All gov sites are targeted. That's not evidence of a vulnerability. My company would never produce a commissioned report using this level of evidence.
ReplyDeleteLove how your comments and opinion keeps changing when you meet resistance. Anyways.. Good for you and your company :-) enjoy.
ReplyDeleteThis comment has been removed by the author.
DeleteI don't know what that means, Dave. I pointed out a few inferior sources. You responded with more of the same, and threw in a ridiculous question about Christians for good measure. Those sources simply claimed that the site was targeted. Have you seen the statistics for government websites that are targeted? You might as well have produced a report that said deer are targeted by hunters.
DeleteI'm not sure you're the best source for specification of evidence Mr. Carr-
ReplyDeletehttp://thediplomat.com/2010/10/was-china-behind-stuxnet/
Stuxnet developed from China:
"The reason why is that if you look at the states that have been impacted—it has generally been those in Asia or Eurasia—what they have in common is that they are producers of key resources."
Where's the data to back this claim up? Just like the old saying- hash or it didn't happen.
While my conclusion was wrong, the sources that I used were sound. It certainly wasn't Popular Mechanics, the Daily Caller, and the Heritage Foundation. Open source investigation relies at the very least upon quality of evidence. Whether the conclusions drawn from that evidence are right or wrong is a different matter.
DeleteFirst, I see nothing unethical about what is reported in the TrustedSec report.
ReplyDeleteAs to the first part of the report -- the review of things published -- these aren't all lies. I am the author of one of those blogs: http://blog.isthereaproblemhere.com/search/label/Healthcare.gov . I have reported details to back up many of my concerns. (And some details I've not published or delayed publishing until fixed.) I understand that others may disagree as to the severity of some of the issues. CMS has confirmed that some of what I reported is so -- and they've been fixing others they haven't publicly addressed.
Regardless of the political leanings of the Heritage Foundation, I have seen/heard many reports that CMS has confirmed the claims of the original Heritage report.
User enumeration can be done on many sites. However, it can be prevented by not revealing usernames or email addresses. Given the nature of the data collected by Healthcare.gov, this should be prevented. In the case of Healthcare.gov, it revealed all of the following without authentication: usernames, email addresses, password reset codes, and security questions. These vulnerabilities existed for many weeks after release before being fixed. The site no longer reveals some of these, but some issues remain. Revealing this information in the first place was irresponsible on the part of the developers and CMS.
I partly agree with the anonymous government employee. The issues reported in regards to data.healthcare.gov are likely not things that should cause concern. That is a public website -- one that, as I understand, doesn't contain information that needs to be protected from public view.
Thanks for your comment, Ben. Your contribution was closer to what I would expect to see in a report about known vulnerabilities. Unfortunately that wasn't true for much of the open source research in that report. Additionally, they failed to do what you did in your own blog, which was to indicate which of the reportable issues had been fixed.
ReplyDeleteAnd while the Heritage Foundation information was correct, there were better sources for the same information which weren't as politically biased. Had the authors weighted their sources for political bias and looked for more non-biased reporting of that problem they could have had the same effect (reporting a valid flaw) without the criticism of using a biased source.
Thank you as well for acknowledging at least the data.healthcare.gov point that the anonymous gov't employee made rather than simply posting a blanket dismissal of his objections.
Yes, I too would have expected different and/or more sources to be used for a number of the items in the report. It is important to weigh a variety of sources and make some effort to understand any political motivations -- real or likely to be perceived by one's audience. Even when they are correct, political organizations aren't the most credible witnesses.
DeleteGiven the format and manner in which this report was released, I would have expected better quality in the analysis and reporting. It seemed to lack focus and contained a significant volume of content that distracts from the issues that should be getting attention. It reads more like a rambling blog post (or blog comment) than a formal report.
That said, the government employee's response is concerning. It attacks the reporter and a few misunderstandings rather than acknowledging or addressing the specific concerns. We should not demand others provide reproducible proof before we consider their concerns. No matter how poorly reported, I believe we should all seriously consider reports that question the security of the information systems we build, test, maintain, and/or operate. Even when concerns are unfounded, investigating the concerns and articulating a response can help us better understand our systems.
Ben - thanks for the response - on your response, I was asked to provide an opinion on the security and keep politics on it. There was several issues I wasn't able to expand upon because of the risk. The first, the ability to enumerate over a 100,000 individuals email addresses, first name, last name, and other personal information. The second the ability to upload and have an individual execute malicious code amongst other things. These details still can't be provided as they have not been fixed or addressed. The purpose of the post was to say as a security researcher, I believe the site is vulnerable and subjective to attack and that's my opinion. While I respect your opinion, the ramblings of a blog post are inaccurate as I was tasked with providing a written testimony of my research and opinion on the matter as it pertains to security - nothing more and nothing less. Additionally, I'm not able to "hack" the site, the user enumeration flaw would typically be a low risk item for me on a normal assessment, simply changing the response to invalid username/and or password would alleviate this. It's a much more symbolic fact that all of these exposures together were released without formal testing which is indicative of a lack of formal security testing or integration with security. This is coming to light more and more everyday to what our research put together.
DeleteLike I said before in previous posts, as to not impact and expose sensitive information of others, many of the more serious ones that haven't been addressed have been removed - as soon as they are addressed, would be more than happy to share.
Dave:
DeleteI understand the inability to provide details on some issues. Sometimes we have to be vague because it wouldn't be wise to disclose details until after the issue is fixed.
I also understand that you've not "hacked" the site. There are a number of tests I'd do (and I suspect you'd do) if granted permission to try to exploit the known and not-known vulnerabilities. Without that permission, ethics and the law constrain us to only looking at what is publicly exposed.
My criticism of the report is more on the presentation than the security concerns raised. My characterization of it as "like a rambling blog post" was too harsh -- not what I intended. I was attempting to contrast a formal report to the sorts of things I often blog. I would have preferred to have seen a more concise executive summary and more detail on the actual issues. My intent was, in response to the anonymous government employee's response, to advocate for better advocacy in our reporting than to attack your report. Most of the issues you have raised (both those fixed and those not yet addressed) should be of concern to all of us.
I agree with your overall assessment that the many issues -- many that seem rather benign on their own -- are symbolic of bigger problems. There appears to have been a lack of care for security -- in design of components, in integration, and in testing.
And, so far the responses from government employees -- both the anonymous and those officially responding -- are mostly failing to address the concerns that you, I, and others have raised.
Should have read "keep politics out of it".
ReplyDeleteI think my concern with all this has been the shut it down mantra. I would tend to agree that it probably sucks, why? Because if they didn't do load testing they probably missed a lot of other things. But from the information given saying the plug needs to be pulled doesn't help anyone. If you believe that they should allow an outside audit I am fine with that. IMHO information security has people in it who are extremely alarmist (I put myself in that category as well). To a certain extent this can be good, but it also can harm credibility as well. Example would be showing the number of attacks on a website. What is an attack? Is it an NMAP scan? If it is an NMAP scan then you will get thousands per day. Is it SQL Slammer bots still trawling the web for SQL boxes listening on the web that haven't been patched in years? If it is you will get thousands per day. Would I classify them as really valid attacks? Depends on if I am trying to sell something :-). It could be a simple example of differences in measurement we have no idea but to compare the number of "attacks" between two sites when we haven't even normalized definitions doesn't really help. I have seen these reports enough to understand that 1) When you start to dig into them there tends to be a lot of false positives not that this is a bad thing, it is just that they shouldn't be treated as true issues until they have been verified and 2) I know that you really can't do a proper test unless you either break the law or have permission. I am of the opinion that doing a passive scan is fine. I have no issue with that. It is then taking that information and claiming it as gospel and yelling the sky is falling. If your report had been we believe we have some issues and are asking to get access to do a proper report I would have no problems. I also keep hearing this 500 million LOC code metric and to that I am calling BS. If we assume that 1000 devs worked for about 3 years with no time off we get about 500 LOC/day. No way is that possible. This again shows the issue though we are screaming about something we only have hearsay knowledge about. These are just my two cents.
ReplyDeleteI think my concern with all this has been the shut it down mantra. I would tend to agree that it probably sucks, why? Because if they didn't do load testing they probably missed a lot of other things like security. I also wouldn't look at something from the outside, read some hearsay and claim to be an expert on the security of what I wasn't really able to properly evaluate. From the information given saying the plug needs to be pulled doesn't help anyone. If you believe that they should allow an outside audit I am fine with that.
ReplyDeleteIMHO information security has people in it who are extremely alarmist (I put myself in that category as well). To a certain extent this can be good, but it also can harm credibility as well. Example would be showing the number of attacks on a website. What is an attack? Is it an NMAP scan? If it is an NMAP scan then you will get thousands per day. Is it SQL Slammer bots still trawling the web for SQL boxes listening on the web that haven't been patched in years? If it is you will get thousands per day. Would I classify them as really valid attacks? Depends on if I am trying to sell something :-). It could be a simple example of differences in measurement we have no idea but to compare the number of "attacks" between two sites when we haven't even normalized definitions doesn't really help. I have seen these reports enough to understand that 1) When you start to dig into them there tends to be a lot of false positives not that this is a bad thing, it is just that they shouldn't be treated as true issues until they have been verified and 2) I know that you really can't do a proper test unless you either break the law or have permission. I am of the opinion that doing a passive scan is fine. I have no issue with that. It is then taking that information and claiming it as gospel and yelling the sky is falling particularly on TV. If your report had been we believe we have some issues and are asking to get access to do a proper report I would have no problems.
I also keep hearing this 500 million LOC code metric and to that I am calling BS. If we assume that 1000 devs worked for about 3 years with no time off we get about 500 LOC/day. No way is that possible. This again shows the issue though we are screaming about something we only have hearsay knowledge about. These are just my two cents.
Thanks, Dreamspider. You make a lot of excellent points regarding the LOC metric, the importance of defining exactly what an "attack" is and most of all, the need to have a proper security audit done. Obviously I'm a critic of the way a "passive" test was used in this case.
ReplyDelete