Sunday, November 24, 2013

In OSINT, All Sources Aren't Created Equal

"In evaluating open-source documents, collectors and analysts must be careful to determine the origin of the document and the possibilities of inherent biases contained within the document."
- FM2-22.3: Human Intelligence Collector Operations, p. I-10
"Source and information evaluation is identified as being a critical element of the analytical process and production of intelligence products. However there is concern that in reality evaluation is being carried out in a cursory fashion involving limited intellectual rigour. Poor evaluation is also thought to be a causal factor in the failure of intelligence."
- John Joseph and Jeff Corkill "Information Evaluation: How one group of Intelligence Analysts go about the Task"
These two quotes illustrate the long-running problem that has plagued commercial cyber security reporting for many years. There are very few unclassified OSINT standards of source evaluation and even less for cyber threat intelligence; at least that I could find while doing research for this article. 

The field of cyber intelligence is fairly new and fortunately, thanks to the Software Engineering Institute at Carnegie Mellon and the work of Jay McAllister and Troy Townsend, we can take a credible look at the state of the practice of this field:
"Overall, the key findings indicate that organizations use a diverse array of approaches to perform cyber intelligence. They do not adhere to any universal standard for establishing and running a cyber intelligence program, gathering data, or training analysts to interpret the data and communicate findings and performance measures to leadership."
- McAllister and Townsend, The Cyber Intelligence Tradecraft Project
The one thing that isn't covered in their report is the issue of source validation and how that contributes to the validity or value of the intelligence data received. However they did write a follow-up white paper with Troy Mattern entitled "Implementation Framework - Collection Management (.pdf)" 

Please take some time to study the framework and read the white paper. It's an ambitious and very thorough approach to helping companies understand how to get the most value from their cyber intelligence products. Unfortunately, while it specifies data evaluation and source validation, it doesn't provide any specific guidelines on how to implement those two processes.

Fortunately, there has been some great work done on source analysis for Human Intelligence (HUMINT) that I believe can be applied to Cyber intelligence and OSINT in general. It's a paper written by Pat Noble, an FBI intel analyst who did his Masters work at Mercyhurst University's Institute for Intelligence Studies: "Diagnosing Distortion In Source Reporting: Lessons For HUMINT Reliability From Other Fields"

A PowerPoint version of Noble's paper is also available. Here are a few of the slides from that presentation:

We recognize these failings when it comes to human intelligence collection but for some reason we don't recognize them or watch for them when it comes to OSINT. The crossover application seems obvious to me and could probably be easily implemented. 

I started this article with a quote from the Army Field Manual FM2-22.3: Human Intelligence Collector Operations (.pdf). Appendix B in that manual contains a Source and Information Reliability Matrix which I think is also applicable to Cyber intelligence or any analytic work that relies upon open sources.

I think a graph like this could be applied with very little customization to sources referenced in cyber intelligence reports or security assessments produced by cyber security companies. 

The West Australian Police Force study by John Joseph and Jeff Corkill "Information Evaluation: How one group of Intelligence Analysts go about the Task" recommended the use of the Admiralty Scale which is identical to the Army's matrix shown above:

Again, these scales were developed to evaluate human sources, not published content, but they certainly seem applicable with some minor tweaking. 

It's important to note that only part of the problem lies in the lack of source evaluation methods. Another very large contributing problem is the lack of standardized cyber intelligence tradecraft pointed out by McAllister and Townsend in their Cyber Intelligence Tradecraft paper:
"Tradecraft: Many government organizations have adopted the intelligence community standard of consistently caveating threat analysis with estimative language and source validation based on the quality of the sources, reporting history, and independent verification of corroborating sources. Numerous individuals with varying levels of this skillset have transitioned to cyber intelligence roles in industry and academia, but the practice of assessing credibility remains largely absent. The numerous analytical products reviewed for the CITP either did not contain estimative or source validation language, or relied on the third-party intelligence service providing the information to do the necessary credibility assessment." (p.11)
And of course due to the newness of the field there's no standard yet for Cyber Intelligence training (McAllister and Townsend, p. 13). 


There are numerous examples of cyber security reports produced by commercial and government agencies where conclusions were drawn based upon less than hard data, including ones that I or my company wrote. Unless you're working in a scientific laboratory, source material related to cyber threats is rarely 100% reliable. Since no one is above criticism when it comes to this problem, it won't be hard for you to find a report to critique. In fact, it seems like a different information security company is issuing a new report at least once a month if not once a week so feel free to pick one at random and validate the sources using any of the resources that I compiled for this article. 

If you know of other source evaluation resources, please reference them in the comments section. 

If you're a consumer of cyber intelligence reports or threat intelligence feeds, please ask your vendor how his company validates the data that he's selling you, and then run it through your own validation process using one of the tools provided above. 

I'd love to hear from any readers who implement these suggestions and have experiences to share, either in confidence via email or in the comments section below.

UPDATE (11/24/13): A reader just recommended another excellent resource: Army Techniques Publication 2.22-9 "Open Source Intelligence". It discusses deception bias and content credibility, both of which must be accounted for in source validation.

Thursday, November 21, 2013

U.S. Gov Employee Responds to TrustedSec's Review of

After I wrote yesterday's article "The Questionable Value and Ethics of TrustedSec's Pen Test of the Website", I received an email from a well-respected employee of a large government agency who had read TrustedSec's report on the website. This employee has asked me if I would publish the content of that email on my blog. Here it is with some minor formatting changes.


So let's put aside the isc2 ethics violation by TrustedSec that this "report" is and instead focus upon its content."

The report is split into two parts, one based upon public open source intel gathering, and on upon actual "analysis". Contrary to what Goebbels might say, repeating a lie does not make it true. The first half of the "analysis" consists of misquotes and out of context statements about news reports, blog postings and the Heritage foundation (an anti-Affordable Care Act org). 

They extrapolate from news articles and jump to conclusions that would be laughed out of a Bsides conference, let alone a court of law. Most of the "observations" are generic in nature with no supporting detail. Everything is anecdotal. Everything is hearsay. There is no direct observation of any vulnerability, and only "potential risks". 

Many of the articles highlight pre-launch issues that have since been resolved, and others are issues common to most web application (hello, user enumeration? Seriously? Any site with a unique user account has this issue).

This lack of substance extends to the second part of the "analysis" which shows a lack of understanding of both what is and what security is. 

In the professional world of cyber security there are two concept at the heart of computer forensics; peer review and reproducibility. Professionals understand that their word is not enough and they actually have to show something that the community and their peers can reproduce. None of their findings are "reproducible" vulnerabilities. They are all vague possible-maybe-there-could-be risks, or worse yet, a gross misunderstanding of what they are "analyzing."

They raise issues with things like the Terms of Service (TOS).

They raise issues with is not just a website, it is a complex node in a web of Federal, State, and private systems that interconnect to produce the site. The data in it comes from state exchanges, medicare, the IRS, SSA, and other Federal/state agencies, plus private insurers. It's not just a webserver/webapp with a back end database like something circa 2003.
They raise an issue that data will be shared with outside agencies which shows they don't understand what is. Then they raise another issue about public profiles on the site. The fact is that is an open data initiative based on the data gathered from insurers. Public profiles are a feature, not a bug, of that SEPARATE platform.

These two examples show the lack of due care conducted on this analysis. Please take a moment to read the "results" [CARR: A link to TrustedSec's report is provided below]. The level of writing and actual deliverable are so laughable that if a contractor had produced this for my agency I would have terminated their contract on the spot. (The report shows) no due diligence, sloppy work, and worst of all it is wrong in its "conclusions". 

Determinations need proof beyond media quotes and theoretical issues. They need to be based in fact.


Here's a link to TrustedSec's public report (.pdf) for those readers who wish to review it and assess the above criticism for themselves. Comments are open.

UPDATE (12/13/13): "On December 11, in order to address ongoing questions, Committee members and staff received a classified briefing from Dr. Kevin Charest, the HHS Chief Information Security
Officer, and Ned Holland, HHS Assistant Secretary for Administration. Portions of this briefing
were classified to protect information relevant to national security. This memo contains a summary of the unclassified portion of the briefing."

Friday, November 15, 2013

Russian Venture Capital (RVC): A Report on Funding Priorities and RF Government Affiliations

Taia Global regularly produces custom reports on foreign research and development activities in Russia and China. Our most recent report examines Russian Venture Capital (RVC), an Open Joint Stock company (OAO RVC) with initial funding from the Investment Fund of Russia through the Federal Agency for STate Property Management (Rosimuschestvo). It's charter allows RVC to invest both domestically and overseas. RVC's Board of Directors limited investments by RVC to companies with products on the Russian government's critical technologies list.

This report is 17 pages long with graphics and two appendices, including the above-mentioned critical technologies list. We examined the background of RVC's executives as well as the firm's investments and its U.S. affiliations.

We are offering this report for a limited time to non-subscribers for $225. Interested parties may order via this link or by calling (855) 877-8242.

Friday, November 1, 2013

Level 3 Communications, the NSA, and the end of the Physical-Digital Divide. What needs to be done?

The Level 3 Communications (NYSE: LVLT) blog recently published an article entitled "Say Goodbye to the Physical-Digital Divide." It's a light-hearted, upbeat corporate feel-good piece about how television shows are become Twitter-enabled. It's also a very disturbing piece when you realize that Level 3 is one of the Tier 1 backbone providers who has assisted the NSA in its collection efforts:
This is an exciting time!  Not only for Joe Consumer, who is being further enabled (and actively encouraged) to merge his offline and online behavior, blurring the lines of the physical-digital divide, but also for major content providers – many of whom we’re fortunate enough to call customers.  This is the new model of content consumption.  Always-on and always-available. Cross-media and cross-platform. 
Think about that from the standpoint of legal intercepts and data collection, and you'll see my point. We used to be vulnerable based upon what we read at the library, what we threw away in our trash, and what we wrote to our friends. Today, that has expanded exponentially and we've lost control of exactly how and where we are vulnerable to exposure.

Now consider that Level 3 is Google's upstream provider. Is that how the NSA was able to intercept the data traveling between Google's data centers? To be clear, Level 3 isn't doing anything illegal, nor is the NSA for that matter. And that's precisely the problem that needs addressing.

In less than 10 years, the physical - digital divide has disintegrated. In less time than it takes a human being to achieve mastery over a skill, technology has exponentially expanded how we interact with each other and, conversely, how we can harm each other.

Intelligence and law enforcement agencies, whose mission is to identify and intercept those who wish to cause us harm, have leveraged legal regimes like the Patriot Act, EO 12333, etc. to gain a foothold within the networks that are the primary supports (i.e., backbone) for our digital environment. The difference between what those out-dated laws still allow and what technology has made possible in the way of data collection and analysis is where our focus needs to be. In other words, the laws must be amended to catch up with how exposed we are in today's digital and physical world so that a better privacy:security balance can be restored.

Wasting time bashing the NSA and other intelligence services does more harm than good because it fails to address the real problem (out-dated authorities that need revising) in favor of lashing out at an easy and unpopular target - the NSA and its fellow agencies who diligently attempt to accomplish the very difficult tasks that we expect from them.

In an effort to help move this debate forward and clarify where reforms are needed, I've set aside two hours for a panel discussion at Suits and Spooks DC on how our parallel needs for security and privacy can be met through reform of the current laws authorizing data collection by the IC. It's not an easy panel to fill, so let me know if you have any suggestions for experts to participate on it. Dr. Catherine Lotrionte of Georgetown University will be the moderator.