Technology


3ders.org – Jay Leno uses 3D scanner and 3D Printer to replace old …

May 18, 2012 by (author unknown)


Jay Leno’s biggest hobby is car collection, he has a lot of old cars, the only problem is he can nowhere find a spare replacement part. But Jay and all the guys at …
www.3ders.org/…/20120514-jay-leno-uses-3d-scanner-and-3d…


App Stores = Markets; Cloud Platforms = Customer Ecosystems

May 15, 2012 by Patty Seybold


There’s clearly a stampede going on to combine app stores, digital downloads, and cloud storage/back-up as ways to create competing and coopetive ecosystems. Amazon, Apple, Facebook, Google, Microsoft (which just invested in Barnes and Noble’s Nook e-Readers) all now have such similar business models and ecosystem strategies, it’s no longer surprising when one of them makes the next move. Their goal: create a comfy “place” for us to hang out and DO everything we need to do in order to live our digitally-connected lives, from downloading music and videos and books, to chatting with our friends, to keeping our important digital assets, email and files backed up, to strutting our stuff and selling our wares.

In his Forbes article, entitled Facebook’s App Store Heats Up Convergence of Big 5, Haydn Shaughnessy writes:

“Facebook’s App Center will be a mega-shop for Apps from all operating systems, with a rating system based around engagement. Facebook will not take a cut if it directs people to iOS or Android but will be encouraging developers to do more for Facebook, where it will take 30%.

The most remarkable thing to note is the large areas of convergence. Underlying all these developments is, in fact, a converged business model: platform and ecosystems. Skimming this morning’s news reports nobody is questioning Facebook’s ability to sell apps or to replicate Apple’s success as a platform and ecosystem business. Platform and ecosystem businesses seem to be uniquely able to launch major new initiatives, with ease.

Right now the top 5 or six American companies in these respective domains [Amazon, Apple, Facebook, Google and Microsoft] are working their way towards a new corporate form, a new way of organising resources and creating wealth. It pays to take note of it. Platforms and ecosystems will rule.”

~ Haydn Shaughnessy, Forbes Contributor and co-author of
The Elastic Enterprise: The New Manifesto for Business Revolution

It’s true that there’s usually a convergence between these 5 players’ cloud platforms and their app markets. But I believe that it’s important to point out that an App Market or Store is “just” a marketplace. It’s a convenient place to go to find and buy something. That’s not unimportant, but it’s not as compelling as a customer ecosystem that is designed to help us get things done, beyond buying things. We believe that true customer ecosystems probably require cloud capabilities—so we never have to worry about losing “our stuff.” But, in order to truly win the hearts and minds of customers, customer ecosystems also need to make it easy for us to manage our stuff and to get things done. It’s not clear to me that the Facebook ecosystem is doing that yet.


Augmented reality and location services drive growth in MEMS

May 6, 2012 by IDG News Service


The growth of new mobile technologies such as augmented reality and location-based services is driving adoption of sensor devices that can detect environmental conditions such as speed, temperature and direction. The growth of new mobile technologies …
See all stories on this topic »


Developer Week in Review: Are APIs intellectual property?

May 4, 2012 by James Turner


Returning after a brief hiatus due to my annual spring head cold, welcome back to your weekly dose of all things programming. Last week, I was attending the Genomes, Environments and Traits conference (I’m a participant in the Personal Genome Project), when I got notified that WWDC registration had opened up. I ended up having to type in my credit card information on my iPhone while listening to the project organizers discuss what they were doing with the saliva I had sent them. The conference itself was very interesting (although I was coming down with the aforementioned cold, so I wasn’t at the top of my game). The cost to sequence a genome is plummeting — it’s approaching $1,000 a pop — and it has the potential to totally revolutionize how we think about health care.

It’s also an interesting example of big data, but not how we normally think about it. An individual genome isn’t all that big in the scheme of things (it’s about 3GB uncompressed per genome), but there are huge computational challenges involved in relating individual variations in the genome to phenotype variations (in other words, figuring out what variations are responsible for traits or diseases).

While all the West Coast developers who slept through the WWDC registration period lick their wounds, here’s the rest of the news.

APIs are copyrightable, unless they aren’t?

These days, I feel like you need to consider a minor in law to go with your computer science degree. In the latest news from the front, we have conflicting opinions regarding the status of APIs. On the one hand, the judge in the Oracle versus Google lawsuit has instructed the jury they should assume that APIs are copyrightable. As the linked article discusses, this could have ominous implications for any third-party re-implementation of a programming language or other software that is not open source.

Over in Europe, however, a new ruling has stated that programming languages and computer functionality are not copyrightable. So, depending on which side of the ocean you live on, APIs are either open season, or off limits. No word yet as to the legal status of APIs on the Falkland Islands …

Fluent Conference: JavaScript & Beyond — Explore the changing worlds of JavaScript & HTML5 at the O’Reilly Fluent Conference (May 29 – 31 in San Francisco, Calif.).

Save 20% on registration with the code RADAR20

Code to make your head hurt.

For those of you who like to celebrate the perversities of life, it’s hard to beat the International Obfuscated C Competition, which just released its 2011 winners. For your viewing pleasure, we have programs that compute pi, chart histograms, and even judging programs for obfuscation, all written in a manner that will have code reviewers running to the toilet with terminal bouts of nausea.

And speaking of C …

We tend to focus a lot of attention on emerging languages, partially because many of them have novel features, and partially because the grass is always greener in a different language. It’s instructive to step back sometimes and take a look at what people are actually using. The latest TIOBE Programming Community Index, which measures how much code there is out there in each of the various languages, has a new top dog, and it’s our old friend C. In fact, when you factor in C#, C++ and Objective-C, C-related languages pretty much own the category. Java has now fallen to the second position, and you have to go all the way down to sixth place to find a scripting language, PHP.

Importantly, all the hot new languages, like Erlang and Scala, don’t even make the top 20, and you only need half-a-percentage point to get in that list. As much as we like the new darlings on the block, the old veterans still are where most of the action (and money) is.

Got news?

Please send tips and leads here.

Related:


Netflix Never Used Its $1 Million Algorithm Due To Engineering Costs

April 16, 2012 by Casey Johnston, Ars Technica


Netflix awarded a $1 million prize to a developer team in 2009 for an algorithm that increased the accuracy of the company’s recommendation engine by 10 percent. But it doesn’t use the million-dollar code, and has no plans to implement it in the future, Netflix announced on its blog Friday. The post goes on to explain why: a combination of too much engineering effort for the results, and a shift from movie recommendations to the “next level” of personalization caused by the transition of the business from mailed DVDs to video streaming.

Netflix notes that it does still use two algorithms from the team that won the first Progress Prize for an 8.43 percent improvement to the recommendation engine’s root mean squared error (the full $1 million was awarded for a 10 percent improvement). But the increase in accuracy on the winning improvements “did not seem to justify the engineering effort needed to bring them into a production environment,” the blog post said. By that time, the company had moved on anyway.

When Netflix announced the contest to improve the service in 2007, its business was centered on DVDs, which are dealt with by customers in periods of days or weeks and provide little granular data. Now that Netflix’s primary offering is streaming, it has access to much more information: “streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially,” reads the post.

This doesn’t seem like much data, but it broadly affects the full screen of personalized recommendations a user might see on their Netflix home page. If a user begins a movie from “Imaginative Time Travel Movies from the 1980s” but quickly closes it, the homepage could shuffle that category down and place a new category at the prime top position that still speaks to the customer’s watching history; something less sci-fi and more 1980s, or vice-versa.

(Mostly) gone are the days that customers would fill their DVD lists with artsy indie films or all of the Academy Award-winning documentaries they could, only for them to remain in queue purgatory. Netflix Watch Instantly is about the here and now, and Netflix is priming to respond to that time frame.


Matterport's scanner can create a 3D model from anything …

April 10, 2012 by Sarah Mitroff


3D printing is turning heads these days, especially with do-it-yourselfers. Y Combinator startup Matterport has invented a small scanner, pictured above, that can scan any space or object and create a 3D model.
VentureBeat


Apple Invents a Killer 3D Imaging Camera for iOS Devices

April 10, 2012 by Patently Apple


Apple has invented a killer 3D imaging camera that will apply to both still photography and video. The new cameras in development will utilize new depth-detection sensors such as LIDAR, RADAR and Laser that will create stereo disparity maps in creating …
See all stories on this topic »


The Dangers of the Minimal Viable Product

April 9, 2012 by Scott Anthony


A movement originating from the United States’ West Coast has sought to transform the creation of new businesses from an art to a science. The intellectual leader of the movement is Steve Blank, a serial entrepreneur who now teaches at Stanford and the University of California at Berkeley. One of Blank’s disciples, Eric Ries, turned his wildly popular Startup Lessons Learned blog into The Lean Startup, one of 2011′s best business books.

I’m a huge fan of this work and suggest that all innovators study the movement closely. One area that deserves particular attention is the notion of the minimal viable product (MVP). The concept is pretty simple. Because it’s next to impossible to be sure that your idea is good until you bring it into the marketplace, don’t waste time trying to fine-tune a product that is destined to be wrong. Instead, put something “good enough” in the marketplace. Let real customers use the product and learn from their feedback (Henry Mintzberg dubbed this process “emergent strategy” in an influential 1985 article [PDF]).

Sometimes, though, a Minimal Viable Product turns into a Mediocre Value Proposition. A company might introduce a product in the marketplace. A few customers find it interesting (you can always find a few customers). Results fall short of expectations, but the company says, “Well, it’s just a minimal viable product for learning.” The company makes a few tweaks, scales up spending . . . and falls flat on its face.

Part of me wonders if this isn’t what happened to Maghound. In late 2008, while prepping for a presentation to some magazine industry bigwigs, a colleague happened on a new venture that had been recently launched by Time, Inc. — “Netflix for magazines” — where consumers could get different magazines every month without having to have a fixed subscription to those magazines. Sounds great, doesn’t it?

Earlier this year, Maghound shut down. One commentator said that it’s a clear sign that the “Netflix of anything” model doesn’t work. I think it shows the limitations of a “good enough” offering.

Maghound’s catchy idea and relatively simple signup seemed attractive. After all, I often would pick up random magazines at airports and enjoyed flipping through esoteric titles left behind on planes or in doctor’s offices. But after I signed up for the service, it took forever for a title to arrive. Instead of changing titles on the fly, it took weeks before Maghound’s system registered a change so I could get a new magazine. Nor was there anything approximating Netflix’s recommendation system to help steer me to surprising titles I might like. After a few months, I cancelled my Maghound membership.

“Good enough” is a great way to start the innovation journey because it enables learning in the most important laboratory of all — the marketplace. But it’s hard to build a compelling business with something that’s just barely adequate. Customers might be intrigued enough to try it once, but they won’t come back.

How do you make sure your minimal viable product isn’t hiding a mediocre value proposition?

  1. Ensure you have the right expertise on your team. Depending on your industry, that might require excellence in design, programming, chemistry, sales, or dozens of other areas. Raw potential can develop into world-class talent, but the portraits of most successful startups feature some gray-haired people who have “been there and done that.”
  2. Think beyond the product to the full offering and business model. Competitive advantage comes from innovative ways of creating, capturing, and delivering value. Success requires fine-tuning more than features and functions.
  3. Act on the learning. One of the biggest warning signs for a new venture is a significant gap in product releases. An even bigger warning sign is when there isn’t any plan to introduce an upgraded offering. Minimal viability descends quickly into mediocrity if there isn’t an actionable plan to act on the learning.

Building mediocre products has never been easier. Building great products and, more importantly, great businesses, remains challenging. By all means start with something that’s just good enough to garner feedback from the marketplace. Just don’t stop there.


3D replication is name of the game

March 25, 2012 by The Canberra Times


Just weeks ago doctors at a Belgian hospital scanned the badly-damaged jawbone of an elderly patient and fed the image through a 3D printer where a precision laser beam transformed fine titanium powder into a patient-specific implant.
See all stories on this topic »


Clothes-Shopping Made Easier with 3-D Body Mapping

March 16, 2012 by (author unknown)



Kinect for Windows, Bodymetrics and Bloomingdale’s may have just resolved your hunt for the perfect jeans.


Buttons were an inspired UI hack, but now we've got better options

March 7, 2012 by Jenn Webb


If you’ve ever seen a child interact with an iPad, you’ve seen the power of the touch interface in action. Is this a sign of what’s to come — will we be touching and swiping screens rather tapping buttons? I reached out to Josh Clark (@globalmoxie), founder of Global Moxie and author of “Tapworthy,” to get his thoughts on the future of touch and computer interaction, and whether or not buttons face extinction.

Clark says a touch-based UI is more intuitive to the way we think and act in the world. He also says touch is just the beginning — speech, facial expression, and physical gestures are on they way, and we need to start thinking about content in these contexts.

Clark will expand on these ideas at Mini TOC Austin on March 9 in Austin, Texas.

Our interview follows.

Are we close to seeing the end of buttons?

josh_clark.pngJosh Clark: I frequently say that buttons are a hack, and people sometimes take that the wrong way. I don’t mean it in a particularly negative way. I think buttons are an inspired hack, a workaround that we’ve needed just to get stuff done. That’s true in the real world as well as the virtual: A light switch over here to turn on a light over there isn’t especially intuitive, and it’s something that has to be learned, and re-learned for every room we walk into. That light switch introduces a middle man, a layer of separation between the action and the thing you really want to work on, which is the light. The switch is a hack, but a brilliant one because it’s just not practical to climb up a ladder in a dark room to screw in the light bulb.

Buttons in interfaces are a similar kind of hack — an abstraction we’ve needed to make the desktop interface work for 30 years. The cursor, the mouse, buttons, tabs, menus … these are all prosthetics we’ve been using to wrangle content and information.

With touchscreen interfaces, though, designers can create the illusion of acting on information and content directly, manipulating it like a physical object that you can touch and stretch and drag and nudge. Those interactions tickle our brains in different ways from how traditional interfaces work because we don’t have to process that middle layer of UI conventions. We can just touch the content directly in many cases. It’s a great way to help cut through complexity.

The result is so much more intuitive, so much more natural to the way we think and act in the world. The proof is how quickly people with no computing experience — people like toddlers and seniors — take so quickly to the iPad. They’re actually better with these interfaces than the rest of us because they aren’t poisoned by 30 years of desktop interface conventions. Follow the toddlers; they’re better at it than we are.

So, yes, in some contexts, buttons and other administrative debris of the traditional interface have run their course. But buttons remain useful in some contexts, especially for more abstract tasks that aren’t easily represented physically. The keyboard is a great example, as are actions like “send to Twitter,” which don’t have readily obvious physical components. And just as important, buttons are labeled with clear calls to action. As we turn the corner into popularizing touch interactions, buttons will still have a place.

Mini TOC Austin — Being held March 9, 2012 — right before SXSW — O’Reilly Tools of Change presents Mini TOC Austin, a one-day event focusing on Austin’s thriving publishing, tech, and bookish-arts community.

Register to attend Mini TOC Austin

What kinds of issues do touch- and gesture-oriented interfaces present?

Josh Clark: There are issues for both designers and users. In general, if a touchscreen element looks or behaves like a physical object, people will try to interact with it like one. If your interface looks like a book, people will try to turn its pages. For centuries, designers have dressed up their designs to look like physical objects, but that’s always just been eye candy in the past. With touch, users approach those designs very differently; they’re promises about how the interface works. So designers have to be careful to deliver on those promises. Don’t make your interface look like a book, for example, if it really works through desktop-like buttons. (I’m looking at you, Contacts app for iPad.)

So, you can create really intuitive interfaces by making them look or behave like physical objects. That doesn’t mean that everything has to look just like a real-world object. Windows Phone and the forthcoming Windows 8 interface, for example, use a very flat tile-like metaphor. It doesn’t look like a 3-D gadget or artifact, but it does behave with real-world physics. It’s easy to figure out how to slide and work the content on the screen. People figure that stuff out really quickly.

The next hurdle — and the big opportunity for touch interfaces — is moving to more abstract gestures: two- and three-finger swipes, a full-hand pinch, and so on. In those cases, gestures become the keyboard shortcuts of touch and begin to let you create applications that you play more than you use, almost like an instrument. But wait, here I am talking about abstract gestures; didn’t I just say that abstractions — like buttons — are less than ideal? Well, yeah, the trouble is you don’t want to have the overhead of processing an interface, of thinking through how it works. The thing about physical abstractions (like gestures) versus visual abstractions (like buttons) is that physical actions can be absorbed into muscle memory. That kind of subconscious knowledge is actually much faster than visual processing — it’s why touch typists are so much faster than people who visually peck at the keys. So, once you learn and absorb those physical actions — a two-finger swipe always does this or that — then you can actually move really quickly through an interface in the same way a pianist or a typist moves through a keyboard. Intent fluidly translated to action.

But how do you teach that stuff? Swiping a card, pinching a map, or tapping a photo are all based on actions we know from the physical world. But a two-finger swipe has no prior meaning. It’s not something we’ll guess. Gestures are invisible with no labels, so that means they have to be taught.

Screenshot from Apple's built-in trackpad tutorial
Screenshot from Apple’s trackpad tutorial.

In what ways can UI design alleviate these learning issues?

Josh Clark: Designers should approach this by thinking through how we learn any physical action in the real world: observation of visual cues, demonstration, and practice. Too often, designers fall back on instruction manuals (iPad apps that open with a big screen of gesture diagrams) or screencasts. Neither are very effective.

Instead, designers have to do a better job of coaching people in context, showing our audiences how and when to use a gesture in the moment. More of us need to study video game design because games are great at this. In so many video games, you’re dropped into a world where you don’t even know what your goal is, let alone what you’re capable of or what obstacles you might encounter. The game rides along with you, tracking your progress, taking note of what you’ve encountered and what you haven’t, and giving in-context instruction, tips, and demonstrations as you go. That’s what more apps and websites should do. Don’t wait for people to somehow find a hidden gesture shortcut; tell people about it when they need it. Show an animation of the gesture and wait for them to copy it. Demonstration and practice — that’s how we learn all physical actions, from playing an instrument to learning a tennis serve.

How do you see computer interaction evolving?

Josh Clark: It’s a really exciting time for interaction design because so many new technologies are becoming mature and affordable. Touch got there a few years ago. Speech is just now arriving. Computer vision with face recognition and gesture recognition like Kinect are coming along. So, we have all these areas where computers are learning to understand our particularly human forms of communication.

In the past, we had to learn to act and think like the machine. At the command line, we had to write in the computer’s language, not our own. The desktop graphical user interface was a big step forward in making things more humane through visuals, but it was still oriented around how computers saw the world, not humans. When you consider the additions of touch, speech, facial expression, and physical gesture, you have nearly the whole range of human (and humane) communication tools. As computers learn the subtleties of those expressions, our interfaces can become more human and more intuitive, too.

Touchscreens are leading this charge for now, but touch isn’t appropriate in every context. Speech is obviously great for the car, for walking, for any context where you need your eyes elsewhere. We’re going to see interfaces that use these different modes of communication in context-appropriate combinations. But that means we have to start thinking hard about how our content works in all these different contexts. So many are struggling just to figure out how to make the content adapt to a smaller screen. How about how your content sounds when spoken? How about when it can be touched, or how it should respond to physical gestures or facial expressions? There’s lots of work ahead.

Are Google’s rumored heads-up-display glasses a sign of things to come?

Josh Clark: I’m sure that all kinds of new displays will have a role in the digital future. I’m not especially clever about figuring out which technology will be a huge hit. If someone had told me five years ago that the immediate future would be all about a glass phone with no buttons, I’d have said they were nuts. I think both software and context and, above all, human empathy make the difference in how and when a hardware technology becomes truly useful. The stuff I’ve seen of the heads-up-display glasses seems a bit awkward and unnatural. The twitchy way you have to move your head to navigate the screen seems to ask you to behave a little robot-like. I think trends and expectations are moving in an opposite direction — technology that adapts to human means of expression, not humans adapting to technology.

This interview was edited and condensed.

Related:


Neonode 3D touch headed to tablets and phones: Hands-on

March 5, 2012 by SlashGear


By layering several of the optical sensors into the body of the new frame, however, the new tech can track far more than just pressure – it can follow motion of two fingers in 3D space. Neonode only develops the hardware technology, not the software …
See all stories on this topic »
SlashGear


Making Stuff: 3D Printing on Campus

February 23, 2012 by Campus Technology


For example, company co-founder Bre Pettis appeared on a June 2011 episode of The Colbert Report, where host Stephen Colbert had his face scanned by a 3D scanner and a model of his head composed by the printer. That plan went into the Thingiverse …
See all stories on this topic »


Big data in the cloud

February 22, 2012 by Edd Dumbill


Big data and cloud technology go hand-in-hand. Big data needs clusters of servers for processing, which clouds can readily provide. So goes the marketing message, but what does that look like in reality? Both “cloud” and “big data” have broad definitions, obscured by considerable hype. This article breaks down the landscape as simply as possible, highlighting what’s practical, and what’s to come.

IaaS and private clouds



What is often called “cloud” amounts to virtualized servers: computing
resource that presents itself as a regular server, rentable per
consumption. This is generally called infrastructure as a service
(IaaS), and is offered by platforms such as Rackspace Cloud or Amazon
EC2. You buy time on these services, and install and configure your
own software, such as a Hadoop cluster or NoSQL database. Most of the
solutions I described in my Big Data Market Survey can be deployed on
IaaS services.



Using IaaS clouds doesn’t mean you must handle all deployment
manually: good news for the clusters of machines big data
requires. You can use orchestration frameworks, which handle the
management of resources, and automated infrastructure tools, which
handle server installation and configuration. RightScale offers a
commercial multi-cloud management platform that mitigates some of the
problems of managing servers in the cloud.



Frameworks such as OpenStack and Eucalyptus aim to present a uniform
interface to both private data centers and the public
cloud. Attracting a strong flow of cross industry support, OpenStack
currently addresses computing resource (akin to Amazon’s EC2) and
storage (parallels Amazon S3).



The race is on to make private clouds and IaaS services more usable:
over the next two years using clouds should become much more
straightforward as vendors adopt the nascent standards. There’ll be a
uniform interface, whether you’re using public or private cloud
facilities, or a hybrid of the two.



Particular to big data, several configuration tools already target
Hadoop explicitly: among them Dell’s Crowbar, which aims to make
deploying and configuring clusters simple, and Apache Whirr, which is
specialized for running Hadoop services and other clustered data processing systems.



Today, using IaaS gives you a broad choice of cloud supplier, the
option of using a private cloud, and complete control: but you’ll be
responsible for deploying, managing and maintaining your clusters.

Microsoft SQL Server is a comprehensive information platform offering enterprise-ready technologies and tools that help businesses derive maximum value from information at the lowest TCO. SQL Server 2012 launches next year, offering a cloud-ready information platform delivering mission-critical confidence, breakthrough insight, and cloud on your terms; find out more at www.microsoft.com/sql.

Platform solutions

Using IaaS only brings you so far for with big data applications: they handle the creation of computing and storage resources, but don’t address anything at a higher level. The set up of Hadoop and Hive or a similar solution is down to you.

Beyond IaaS, several cloud services provide application layer support for big data work. Sometimes referred to as managed solutions, or platform as a service (PaaS), these services remove the need to configure or scale things such as databases or MapReduce, reducing your workload and maintenance burden. Additionally, PaaS providers can realize great efficiencies by hosting at the application level, and pass those savings on to the customer.

The general PaaS market is burgeoning, with major players including VMware (Cloud Foundry) and Salesforce (Heroku, force.com). As big data and machine learning requirements percolate through the industry, these players are likely to add their own big-data-specific services. For the purposes of this article, though, I will be sticking to the vendors who already have implemented big data solutions.

Today’s primary providers of such big data platform services are Amazon, Google and Microsoft. You can see their offerings summarized in the table toward the end of this article. Both Amazon Web Services and Microsoft’s Azure blur the lines between infrastructure as a service and platform: you can mix and match. By contrast, Google’s philosophy is to skip the notion of a server altogether, and focus only on the concept of the application. Among these, only Amazon can lay claim to extensive experience with their product.

Amazon Web Services

Amazon has significant experience in hosting big data processing. Use of Amazon EC2 for Hadoop was a popular and natural move for many early adopters of big data, thanks to Amazon’s expandable supply of compute power. Building on this, Amazon launched Elastic Map Reduce in 2009, providing a hosted, scalable Hadoop service.

Applications on Amazon’s platform can pick from the best of both the IaaS and PaaS worlds. General purpose EC2 servers host applications that can then access the appropriate special purpose managed solutions provided by Amazon.

As well as Elastic Map Reduce, Amazon offers several other services relevant to big data, such as the Simple Queue Service for coordinating distributed computing, and a hosted relational database service. At the specialist end of big data, Amazon’s High Performance Computing solutions are tuned for low-latency cluster computing, of the sort required by scientific and engineering applications.


Elastic Map Reduce

Elastic Map Reduce (EMR) can be programmed in the usual Hadoop ways, through Pig, Hive or other programming language, and uses Amazon’s S3 storage service to get data in and out.

Access to Elastic Map Reduce is through Amazon’s SDKs and tools, or with GUI analytical and IDE products such as those offered by Karmasphere. In conjunction with these tools, EMR represents a strong option for experimental and analytical work. Amazon’s EMR pricing makes it a much more attractive option to use EMR, rather than configure EC2 instances yourself to run Hadoop.

When integrating Hadoop with applications generating structured data, using S3 as the main data source can be unwieldy. This is because, similar to Hadoop’s HDFS, S3 works at the level of storing blobs of opaque data. Hadoop’s answer to this is HBase, a NoSQL database that integrates with the rest of the Hadoop stack. Unfortunately, Amazon does not currently offer HBase with Elastic Map Reduce.

DynamoDB

Instead of HBase, Amazon provides DynamoDB, its own managed, scalable NoSQL database. As this a managed solution, it represents a better choice than running your own database on top of EC2, in terms of both performance and economy.

DynamoDB data can be exported to and imported from S3, providing interoperability with EMR.

Google

Google’s cloud platform stands out as distinct from its competitors. Rather than offering virtualization, it provides an application container with defined APIs and services. Developers do not need to concern themselves with the concept of machines: applications execute in the cloud, getting access to as much processing power as they need, within defined resource usage limits.

To use Google’s platform, you must work within the constraints of its APIs. However, if that fits, you can reap the benefits of the security, tuning and performance improvements inherent to the way Google develops all its services.

AppEngine, Google’s cloud application hosting service, offers a MapReduce facility for parallel computation over data, but this is more of a feature for use as part of complex applications rather than for analytical purposes. Instead, BigQuery and the Prediction API form the core of Google’s big data offering, respectively offering analysis and machine learning facilities. Both these services are available exclusively via REST APIs, consistent with Google’s vision for web-based computing.

BigQuery

BigQuery is an analytical database, suitable for interactive analysis over datasets of the order of 1TB. It works best on a small number of tables with a large number of rows. BigQuery offers a familiar SQL interface to its data. In that, it is comparable to Apache Hive, but the typical performance is faster, making BigQuery a good choice for exploratory data analysis.

Getting data into BigQuery is a matter of directly uploading it, or importing it from Google’s Cloud Storage system. This is the aspect of BigQuery with the biggest room for improvement. Whereas Amazon’s S3 lets you mail in disks for import, Google doesn’t currently have this facility. Streaming data into BigQuery isn’t viable either, so regular imports are required for constantly updating data. Finally, as BigQuery only accepts data formatted as comma-separated value (CSV) files, you will need to use external methods to clean up the data beforehand.

Rather than provide end-user interfaces itself, Google wants an ecosystem to grow around BigQuery, with vendors incorporating it into their products, in the same way Elastic Map Reduce has acquired tool integration. Currently in beta test, to which anybody can apply, BigQuery is expected to be publicly available during 2012.

Prediction API

Many uses of machine learning are well defined, such as classification, sentiment analysis, or recommendation generation. To meet these needs, Google offers its Prediction API product.

Applications using the Prediction API work by creating and training a model hosted within Google’s system. Once trained, this model can be used to make predictions, such as spam detection. Google is working on allowing these models to be shared, optionally with a fee. This will let you take advantage of previously trained models, which in many cases will save you time and expertise with training.

Though promising, Google’s offerings are in their early days. Further integration between its services is required, as well as time for ecosystem development to make their tools more approachable.

Microsoft

I have written in some detail about Microsoft’s big data strategy in Microsoft’s plan for Hadoop and big data. By offering its data platforms on Windows Azure in addition to Windows Server, Microsoft’s aim is to make either on-premise or cloud-based deployments equally viable with its technology. Azure parallels Amazon’s web service offerings in many ways, offering a mix of IaaS services with managed applications such as SQL Server.

Hadoop is the central pillar of Microsoft’s big data approach, surrounded by the ecosystem of its own database and business intelligence tools. For organizations already invested in the Microsoft platform, Azure will represent the smoothest route for integrating big data into the operation. Azure itself is pragmatic about language choice, supporting technologies such as Java, PHP and Node.js in addition to Microsoft’s own.

As with Google’s BigQuery, Microsoft’s Hadoop solution is currently in closed beta test, and is expected to be generally available sometime in the middle of 2012.

Big data cloud platforms compared

The following table summarizes the data storage and analysis capabilities of Amazon, Google and Microsoft’s cloud platforms. Intentionally excluded are IaaS solutions without dedicated big data offerings.














  Amazon Google Microsoft


Product(s)
Amazon Web Services
Google Cloud Services
Windows Azure

Big data storage
S3
Cloud Storage
HDFS on Azure

Working storage
Elastic Block Store
AppEngine (Datastore, Blobstore)
Blob, table, queues

NoSQL database
DynamoDB1
AppEngine Datastore
Table storage

Relational database
Relational Database Service (MySQL or Oracle)
Cloud SQL (MySQL compatible)
SQL Azure

Application hosting
EC2
AppEngine
Azure Compute

Map/Reduce service
Elastic MapReduce (Hadoop)
AppEngine (limited capacity)
Hadoop on Azure2

Big data analytics
Elastic MapReduce (Hadoop interface3)
BigQuery2 (TB-scale, SQL interface)
Hadoop on Azure (Hadoop interface3)

Machine learning
Via Hadoop + Mahout on EMR or EC2
Prediction API
Mahout with Hadoop

Streaming processing
Nothing prepackaged: use custom solution on EC2
Prospective Search API 4
StreamInsight2 (“Project Austin”)

Data import
Network, physically ship drives
Network
Network

Data sources
Public Data Sets
A few sample datasets
Windows Azure Marketplace

Availability
Public production
Some services in private beta
Some services in private beta

Conclusion

Cloud-based big data services offer considerable advantages in removing the overhead of configuring and tuning your own clusters, and in ensuring you pay only for what you use. The biggest issue is always going to be data locality, as it is slow and expensive to ship data. The most effective big data cloud solutions will be the ones where the data is also collected in the cloud. This is an incentive to investigate EC2, Azure or AppEngine as a primary application platform, and an indicator that PaaS competitors such as Cloud Foundry and Heroku will have to address big data as a priority.

It is early days yet for big data in the cloud, with only Amazon offering battle-tested solutions at this point. Cloud services themselves are at an early stage, and we will see both increasing standardization and innovation over the next two years.

However, the twin advantages of not having to worry about infrastructure and economies of scale mean it is well worth investigating cloud services for your big data needs, especially for an experimental or green-field project. Looking to the future, there’s no doubt that big data analytical capability will form an essential component of utility computing solutions.

Notes:

1 In public beta.

2 In controlled beta test.

3 Hive and Pig compatible.

4 Experimental status.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Related:


Website from Sherwin-Williams Analyzes Images to Create Color Palette

February 10, 2012 by (author unknown)



Here’s a truly useful idea for Sherwin Williams from McKinney. Chip It! analyzes every pixel in an image to match it to one of the paint manufacturer’s 1,500 paint colors, and composites the top 10 into ChipCards.


How social technologies are extending the organization

November 21, 2011 by (author unknown)


Our fifth annual survey on the way organizations use social tools and technologies finds that they continue to seep into many organizations, transforming business processes and raising performance.
Read more on the McKinsey Quarterly >
   Topics:
High Tech
Business Technology
Surveys

Update your Quarterly feed preferences


How To: Crowdsource Your Next Vacation

August 11, 2011 by Maya Swedowsky


Social media has dramatically changed the way we book travel and plan vacations.

According to a Destinations Analyst survey – “The State of the American Traveler” – nearly 35.8% of travelers listen to the opinions of friends, colleagues or relatives when making travel plans. And with the average person having 130 Facebook friends, social media is a natural place for gathering recommendations from our peers.
Using Facebook Questions to Pick a Vacation Destination

This change has been accompanied by the growth in use of social and web tools for planning vacations; social media has become a necessary partner in experiencing the best vacation possible. So what are the most useful tools for planning in advance and while on vacation?


Planning in Advance: TripAdvisor

An informal survey of my team illustrated that trip planning website behemoths like TripAdvisor are still the go-to resource, followed by recommendations from our social network (online and offline). In fact, almost 90% of those surveyed said that they relied on travel websites like TripAdvisor when choosing a hotel.

Founded just over 10 years ago, TripAdvisor was an early adopter of consumer-generated reviews and has amassed quite the archive of feedback from travelers. While the site has added functionalities and features over the years (e.g. connecting to Facebook to read reviews from your network, a mobile application), it is its massive amount of data that has marked TripAdvisor as a credible resource. While it’s nice to be able to get recommendations from those we know and trust versus strangers, having a lot of reviews from strangers is a close second.

The interface may not be as slick or social as emerging travel sites like Gogobot or gtrot, but TripAdvisor gets the job done. And that’s usually what we’re looking for when we book a hotel or flight – the more confident we can feel about our choice and the quicker we can make it, the better.

Last Minute Planning: Mobile

Geo-location and mobile tools become increasingly important once a trip is underway. After we arrive at our destination, we can use tools like Foursquare or the Time Out mobile app to help us navigate unfamiliar territory. Whether reading tips left by other travelers or city guides created by brands or the local travel board, geo-location tools provide us with simple filter we can apply to our foreign surroundings.

Sites to Watch

Hipmunk is a simple service that enables travelers to book hotels and flights. The most compelling functionality of Hipmunk can be found within the flight search – results for flights are plotted out on a Gantt chart that can be sorted by price, duration, arrival and departure times, non-stop, and “agony.” Yes, “agony” really is one of the options – it is defined as “a combination of price, duration and number of stops.” This sorting option not only appeals to the emotions that go hand-in-hand with traveling, but also helps us rationalize a more expensive flight by quantifying the unpleasantness that we’re eager to avoid. Although Hipmunk is lacking in the consumer reviews department for hotels, its flight booking functionality is definitely worth checking out.

Wanderfly is geared toward adventurous travelers – those who have interests like “eco” or “watersports,” but aren’t quite sure where they want to go. After entering a price range, general date range (e.g. mid-January) and selecting your interests, Wanderfly will serve up suggested destinations, accompanied by recommendations on where to stay, what to do, how much it will cost and what your Facebook friends have to say about the destination. Wanderfly is monetizing the site by giving brands the opportunity to curate recommendations for relevant destinations (e.g. Havaianas, the flip flop brand, recommends beach destination that will “soothe your sole”). While Wanderfly’’s unique, exploratory interface and brand sponsorship opportunities are well-executed, the site may not provide enough structure for the average (adult) traveler.

What’s your go-to website, application or resource when planning a vacation?


Social Networks Will Change Product Innovation

March 18, 2011 by Andrei Hagiu


Much is being written about the impact that new communication technologies and channels (blogs, Facebook, Twitter, YouTube) have on traditional marketing. The deeper question is: Will these new communication channels actually force material changes not just in the way companies market their products but in the strategies and operations they use to develop and build those products as well? In my view, the answer is an emphatic yes. It’s another instance of the proverbial medium that changes the content.

The effects on large established companies and start-ups are different.

Let’s consider large companies first. In theory, YouTube, Twitter, Facebook and blogs should make it possible to engage their customers in new and diverse ways. The choice set is larger and the audience is expanded. So this should be good, right? Not necessarily, as illustrated by the mighty challenges encountered even by media companies in transitioning from the off-line print world to the online one.

As it turns out, managing corporate communication and brand diffusion across these new communication channels is quite different from what they were used to. It is very difficult and costly to maintain a unified voice across all channels and to control information flows to the outside world (particularly for technology companies, whose employees are likely to be Twitter or tech blog stars). The company needs to adjust to a 24/7 dialogue with consumers, investors, and other stakeholders.

And this will require changes in product strategy. Companies are under increasing pressure to significantly shorten their product cycles and rely much less on well-choreographed and fully-controlled product releases. In a world of constant information flows and leaks to the outside world, they need to accept that they will have to work with smaller changes at shorter intervals — and that they will be subjected to much more visibility and scrutiny from the outside world.

The good news is that for some companies, this makes it possible to leverage much more useful input from customers. The dark side is increased exposure to negative shocks due to founded or unfounded rumors. Sure, one could argue that these changes in business models are partly driven by other factors (e.g., the advent of truly ubiquitous bandwidth which made cloud-computing models for numerous software applications possible). Nevertheless, it seems undeniable that communication technologies are playing a major role in prompting these changes.

For small companies and start-ups, the considerations are a bit different. On the positive side, new communication technologies and channels have made it easier to get on the map quickly and to create buzz and word-of-mouth. The problem is that start-ups are now subjected to new and not necessarily desirable pressures, which are exclusively related to communication channels. In short, it is the “fear of being left out” syndrome.

The public’s perception of the relative competitive positions of several start-ups in the same field is increasingly determined by a popularity contest in tech blogs and Twitter posts. Sure, some of these new channels contain very well-informed analyses. But as soon as their effect on the public’s and ultimately investors’ perceptions is taken into account, we end up with self-fulfilling prophecies and exacerbated “superstar” effects: a few companies attract all the attention and, therefore, investor financing, while the rest labor in obscurity and, as a result, have fewer chances of success.

This cannot be a good thing, especially with new, difficult, and unproven technologies. Entering a popularity competition too early can be very detrimental to orderly progress. It also makes it hard if not impossible to stay under the radar, which is crucial for start-ups creating disruptive technologies.

To some extent, this is similar to Facebook creating social pressure on people (fortunately, this seems to vary widely by generation!) to participate and share details of their daily activities. Just like one might worry that teenagers spend too much time writing posts about doing and learning and too little time actually doing and learning, investors may be legitimately concerned that companies are spending too much time posturing on social media as opposed to producing useful things.

But fighting against this trend will most likely be a losing upstream battle. One needs to recognize that the new communication channels will become an integral part of the business world — just like Facebook is becoming an integral part of society — and adapt. Recognize that the new channels require not just new marketing campaigns but also reorganizations and rethinking of entire strategies.

Andrei Hagiu is an associate professor in the strategy group at Harvard Business School.


Quora for Business Not Allowed, But You Should Still Monitor and Respond

January 21, 2011 by jeremiah_owyang


Yesterday, on a client call, I was asked if Quora was going to be relevant, and what businesses should do about it.  I’m sure I’m going to get asked this more often, so I’ll put my answers down for all to see.  If you’re not familiar with Quora, it’s a Q&A website where people can ask questions and others can answer and respond.  If you’ve seen LinkedIn or Yahoo Answers or similar features from Community Platforms, and even Get Satisfaction to an extend, you’ll find a theme.  Quora officially launched 6 months ago in a limited beta, and is founded by Facebook’s former CTO with backing by Benchmark capital, all positive signs for company vitality.

What makes Quora Notable:

  1. A Social Network for Questions and Responses. Like a social network, individual are often putting their real identities and offer extensive features for surfacing the ideal responses up, these features are more advanced that the typical ‘vote up or down’ responses.  The service is interesting, as it seeks to surface the most valuable and relevant answers, but I found that many of my questions didn’t get enough responses to warrant usefulness over every interaction.  There’s a unique blend of checks and balances that allow for the question, summaries, and votes for answers to be surfaced, much like a wiki for Q&A.
  2. An Advanced Set of Features That May Confuse New Users. The user interface offers as lot of features, but some which are complicated.  As I started to probe around and ask questions, it wasn’t clear how to create a question (you actually have to put it in the ‘search field’, an action not native to most users).  Secondly it wasn’t clear what the difference was between a question and a post (which is like a blog post).  A few times, I found the service slow to load pages, likely due to the next point.
  3. Technology Influencers are Currently Present, Spurring Growth. A certain type of technology and web influencer is present (see this frequently updated list), giving fuel to their early but rapid growth.   I’ve seen this before.  First with Twitter, Friendfeed and now Quora.  While this group may not be an influencer group of CPG, Retail, and other industries, they are an early indicator of technology adoption.   For example, in his natural penchant to find and evangelize new services, Robert Scoble, has become a poster boy for this product, much how he did for Friendfeed, further bolstering the growth.   To really see if this is the service that will explode, we’ll need to watch how it shapes the SXSW experience, a conference I use as a bell weather for next-generation technologies
  4. Like All Social Sites, It’s at Risk to be Gamed. Quality responses are likely to surface the highest, yet like all voting websites, search engines, and social networks that allow for followers and friends, those with more connections to the network will be able to assert more influence over content, visibility, and ability to ultimately sway others.  The Quora service seeks to balance this out by trying to surface the most relevant content at the top.  Yet, I assert that those with more popularity will always be able to influence their responses to the top, as you can read in this discussion here, here, and of course on Quora itself.  (Update: Later, Robert proved my point as he was lashed out for gaming Quora)

How Businesses Should Respond to Quora:

  • Businesses Accounts Not Allowed…Yet. Now, If you’re involved in supporting your customers in the social web, either you’re a brand manager, or a community manager, your job is to go where customers are online and respond to them.  While Quora doesn’t currently allow business profiles on the site (see how the Mashable account has been suspended) you should expect in the future that this would be a potential revenue stream for Quora to offer ‘buy outs’ for Q&A just as LinkedIn has done in the past.
  • At A Minimum, Monitor The Discussions. While corporate accounts are not currently allowed, personal accounts are, and most people who are in Quora work at a company and you see them answering questions about their company.  Send your Community Managers (you have them, right?) into this emerging Q&A sites as they would other sites, to monitor and respond if questions go awry.  Glean intelligence by creating an excel sheet and creating a list of the top asked questions related to your brand and use to fuel internal discussions around why these questions are asked, and cascade to the appropriate product and service teams to fix. Likely if one customer is asking questions in Quora, it’s an indicator others are too.  The savvy will use this information to identify leads, such as those following the keywords related to your brand.
  • Advanced? Engage by Providing Helpful Responses. The advanced community managers should be responding to questions related to the lifestyle or workstyle of their brand to demonstrate thought leadership and ability to engage in discussions, adding value to the community.  If questions about your product are answered on other websites, leave a summary respond and point to the original repository of correct information.  Tip: I provided just a link to one of my blog posts where the answer was, and some users asked me instead answer in text, then link.  As always,  be sure to indicate where you work and that you are an employee of the company, as transparency gleans trust.  The goal?  To give the most helpful answers so they rise to the top of the question thread.

You can find my Quora account here, where I will ask and answer questions that I’m passionate or need help with.

Update Jan 22: I met the Quora founders last night at the Techcrunch’s award show, Crunchies and asked him point blank “When will there be services for companies” and he told me that isn’t a direct focus right now (as they’ve indicated).  Although he didn’t say it directly, we should assume ‘Corporate’ accounts won’t be available sometime until they have massive consumer adoption, and also because they have considerable funding to tie them over to build a great service.  This further reinforces my recommendations to you above to monitor and respond now.



Gawker Gives Up on Blogging (And That’s a Good Thing!)

December 1, 2010 by Dylan F. Tweney


Gawker founder Nick Denton is one of the most aggressive and successful blog publishers around.

Which is exactly why online news sites should be paying close attention to his new, and public, strategy intended to re-focus his snarky snack-bite empire on hard, original news, video, some aggregation and TV-like timing.

Denton started with a good gut understanding of what works online, and built a federation of blogs that exploit a similar model (low overhead, smart and fast writers, efficient tech and ad sales) to great effect. Gizmodo, Gawker, IO9, Lifehacker are all category-leading blogs, or close to it; some of them (like Gizmodo) are starting to compete with traditional media as well.

But what is more significant, his operation is devoted to objective, metric analysis in a way that no one else is. Every Gawker story publicly displays how many pageviews and how many new visitors to the site it has delivered. In addition, Gawker Media’s Manhattan headquarters has a large screen that prominently displays each writer’s traffic totals — numbers that figure into writers’ bonuses. Those numbers are at the core of the organization.

And carefully watching the data has shaped his strategy more than anything. It’s also led him to be way ahead of the curve in many cases (I’m thinking of the way he started planning, in 2008, for a 40% decline in ad spending, long before the rest of the media were willing to face up to what was coming).

It’s somewhat amazing, then, that he’s willing to share the results of his insights. Lifehacker today hosted his thoughts on where Gawker Media is going in 2011.

Now, this could be a classic head-fake, and perhaps Denton is only publishing this plan to throw the rest of us off. But I don’t think so. I think the advice in here is solid, and he’s publishing it because the odds of his slow-moving competitors actually being able to capitalize on this information are slim to none. Meanwhile, Denton raises his cred among smaller publishers, editors and writers, some of whom may be eager to work for him at some point.

Be that as it may, there are some good pointers in Denton’s roadmap.

Here’s the Cliff’s Notes version:

1. The news drives traffic — and more importantly, brings in new readers/viewers who later become return readers. Or, as Denton puts it, “aggressive news-mongering trumps satirical blogging.” Deadspin grew from a cult blog to a household name with its reporting on Brett Favre’s indiscretions. Gizmodo hired Wired magazine editor Joe Brown to shepherd features and long investigative pieces, including those from former Wired.com and Wired magazine writer Brian Gardner.

2. Aggregation is important for filling in the gaps between a few breakout stories each day. The solution? First, have two types of editorial staff: editors/curators, and reporters/producers/scoopmongers. Second, relegate the reverse-chronological “blog flow” to a sidebar, and make the big stories, not the filler, the center of attention.

3. Having a variety of content is important — even more so as your audience grows and becomes more diverse.

4. Photos, videos, and strong visual presentations work resonate with audiences now, and they don’t have to be full-blown videos to work.

5. You can sell video ads in banner ad spaces.

6. Gawker is being programmed more like a TV network, with time slots for stories and ad campaigns, and less like a newspaper or magazine.

7. Gawker is going after high-end brand advertising, the traditional stronghold of magazine companies like Hearst and Wired.com’s own Condé Nast — and the TV networks. To do that, they’re going to be moving up-market this year, trying to class up the joint while still being Gawker Media. Sponsorships and time-slot campaigns are the key to moving out of the doldrums of low-value, high-inventory web advertising.

Big media better watch out. Denton’s done with blogging; his next target is finding an even more profitable form of new media that blends aspects of blogging, magazine journalism, and TV.

And while that might be scary, Denton’s also showing off things that we are learning here at Wired.com as well. Reporting matters. Breaking news and getting scoops resonates with readers. And once you have that, you don’t have to be first with everything, and if you are late to a story, you can take the time to fill it out with smart commentary or additional, or contrarian, perspectives.

Threat Level and Danger Room are two great examples. Danger Room is a great mixture of original reporting, aggregation and smart commentary. Threat Level excels at original, smart and authoritative reporting on tough subjects, like cyber-crime, security and copyright — holding its own traffic-wise despite not having the velocity of a typical blog.

Denton’s new format will be very interesting to watch, and doubtless we’ll see many sites start to adopt pieces of it — probably right as Denton figures out his next step.

But rejoice, readers, writers and publishers: news is back, even if it doesn’t look like what used to land on your doorstep every morning!

Additional reporting by Ryan Singel

An earlier version of this essay appeared on Dylan Tweney’s website. For more real-time tech news, follow Dylan Tweney and Epicenter on Twitter!