How to Automate Collection Efforts Using RSS Feeds from Craigslist

I recently wrote a “how to” article on how to automate collection efforts from Craigslist and other sources using RSS Feeds.   That article was posted in Sources and Methods this morning.

Full Link here.

State Dept plans to use Business Intelligence to measure “country sentiment”

Many businesses have been using customer sentiment analysis to gauge customer opinions for years.   I noted in my post “Technology Gap Lessons” that it’s time for the US to begin working with new media to fight terrorism at its core ideologies.

State Dept is now working on measuring anti-Americanism opinions through Business Intelligence methods.   Time will tell if it’s effective, but you don’t get improvements on anything unless you measure them.  Good move.

Read the full article here.

The Shrinking Competitive Advantage

Your competitive advantage may be shrinking.    What can you do about it?

Additional information has been trickling in following the Islamic Terrorist attacks in Mumbai.   Reports say:

1. that the attacks were very sophisticated
2. the terrorists may have used Google Earth to plan routes and targets
3. they may have used Blackberry’s
4. they used GPS
5. they could have possibly monitored Twitter  (yes, that’s being reported)
6. they used AK-47’s and grenades
7. they came via the water on a fast boat
8. they were highly trained to hold off so many police for days
9. there were 10, and maybe 15 of them

Peter Buxbaum, writing for ISN Security Watch, writes about Normalizing Unconventional Warfare–making Special Operations the Norm.    The only problem with this viewpoint is that we end up training people and groups who may one day use the training against us.  Like Al Qaeda.  Blowback.  There is no easy answer to this dilemma.   The reality is secrets and methods are hard to keep these days with so much information available to everyone.  Because of this, the competitive advantages of weapons, communication gear, and transportation will continue to shrink.   What’s true in warfare may also be true in your business, and maybe for you personally.  Is your competitive advantage shrinking?  If so, what can you do about it?

Navy SEAL Commander William McRaven, in his book “Spec Ops”, reminds us that there are three basic keys to success of an operation.

1. Plans need to be simple

2. Preparation and Practice need to drilled over the over again

3. Execution of the plan takes courage, speed, and purpose

The most important one in this context is preparation and practice.   This is the hardest and longest road to success.  Whatever you are preparing for, if you practice enough, again and again, you will hold a competitive advantage when it comes time to execute.  Yes, even if your opponent uses Twitter.

Sorry that this isn’t a sexy simple answer you wanted to hear.  But maybe it’s time to get to work training harder. I know I need to be reminded of the basics every so often.

No Place to Hide

In Robert O’Harrow’s book, No Place to Hide (copyright 2005), he outlines the new surveillance society and does a pretty good job of showing how information is used and potentially abused.   ABC news did a show based on the content from the book and it runs about 40 minutes long.   It’s a few years old but still worth watching.

No Place to Hide ABC Part 1 of 4

No Place to Hide ABC Part 2 of 4

No Place to Hide ABC Part 3 of 4

No Place to Hide ABC Part 4 of 4

The Foolishness of Crowds

How wise have the crowds been at predicting and mitigating the economic meltdown? Some have questioned whether or not this is yet another case of “Intelligence failure.” Crowds can be just plain dumb sometimes.  (Think of sheep.)

In James Surowiecki’s book, “The Wisdom of Crowds…,” he argues that collectively small groups of people should exhibit more intelligence than a lone individual. But he presupposes that everyone in the group is actually thinking. Group think, where everyone tends to agree because of emotion or peer pressure, is a big problem and not only can erase the benefit of the group, but also can make it more difficult for the dissenter(s) to be heard because of increased peer pressure.

So how do you mitigate this problem? Several years ago I visited an exhibit at the Ronald Reagan Presidential Library. When Reagan was trying to make a difficult decision be brought in his top advisors and laid out the problem. He then asked his advisors to cast their opinions on what to do. But one detail, they had to share their opinion with the President privately. Nobody else saw it. After collecting all the opinions, which were cast individually and without peer pressure or group think, the President made his own decision.

It takes guts to stand on your own line, and sometimes you’ll get dumped on, but you just might save a life, or a fortune.

The Power of Virtual Teams

Excellent example on how the power of small motivated teams can serve as the antidote to the ills plaguing bloated beauracracies.

Hat tip to IntelFusion.

Matt Burton: “Set Your Analysts Free.”

Great post by Matthew Burton on the value of Open Source. He underscores my own inspiration for Project Grey Goose when he writes about the phenomenal but ultimately tragic Jim Gray rescue effort last year:

Freedom: The Jim Gray Rescue Effort

The Web is fast and free: fast, in the sense that you can make things happen overnight; and free, in the sense that you don’t need the boss’s permission to implement new ideas. This was no more apparent than in the case of Jim Gray.

Jim Gray was a database pioneer based in San Francisco. He disappeared while sailing in the Bay area last January. He was never found.

But his friends, many of whom were Silicon Valley magnates, did their best to try. One of these friends retasked DigitalGlobe imagery satellites to shoot fresh photos over 3,500 of sea. The result was 1400 gigantic images. So coders automatically split the images up into 560,000 smaller ones that could be easily reviewed; someone else contributed a computer program that automatically sharpened all of the images.

Now that they had over a half million pictures, they needed a way to solicit volunteers to look at them. Amazon.com already had a ready-made solution, so the images were posted on a special page that let virtual volunteers cull through these images in search of Gray’s sailboat. 12,000 volunteers reviewed all 560,000 images three times each.

All of that happened in three days. They didn’t find Gray’s sailboat. But it was the largest search party in history, on top of being simply amazing.

The Web’s speed and freedom made it possible. Had this job been given to an organization with a closed network, this never would have happened.

What about your organization’s intranet? Could your IT team split up those images into easy-to-consume portions? Could they write a script that automatically sharpened all 560,000 images? Could they create a tool that lets the rest of your workforce review the images from their own desks? Could your servers handle the load? And could all of that be accomplished in three days? In order to do that, you need to have the technical talent on hand.

Once you have the talent, those people need permission to work freely and create solutions on the fly. Do they have that permission? Or would your org have to submit justification statements, get contract managers to approve personnel reassignments, and run each line of code past the infosec people before finally getting something live?

On the Web, if you have an idea, you just do it. This is a philosophy and capability the Intelligence Community (and any large, data-driven organization) should keep in mind if it wants to fully exploit its data. Analysts will always have too much information to process and too little time in which to do it (and focusing the “open source intelligence” conversation on culling Web content contributes to that problem). To have any hope, they need an IT and bureaucratic infrastructure that will accommodate improvised solutions.

Again: if you want to make the most of open source, get an open network that lets analysts contribute their own material. Put all of your analysts onto one large network instead of letting them work on lots of small ones. Hire some developers and familiarize them with the analysts’ information problems. Then, set them free and let them experiment.

Data Mining Dangers?

The Economist.com recently posted an article on Data Mining in Government agencies.

“Electronic snooping by the state may safeguard liberty—and also threaten it” Link here.

Most of the information about us is created and maintained not in the Government sector, but in the private sector.  Commercial transactions (grocery store purchases, credit cards, travel bookings, etc.) are collected, mined, and often sold to 3rd parties.  It’s a good and healthy exercise to ensure our Gov’t is following the rules when collecting information, but shouldn’t we hold commercial companies accountable too?