Google Developer Day Report
By arungupta on May 31, 2007
I reached San Jose McEnery Convention Center around 9:15am and the crowd started building closer to 9:30am.
The keynote hall opened at 9:30am. A video with different mahsups (primarily using Google Maps) was shown until the keynote started (around 10 minutes late). Here are some of the websites that were shown in the video: map.movin-360.com (seems like a virtual tour of Berlin city), prague360.com (name says it), dudewheresmyusedcar.com (looking for a used car near a zip code), wepoco.com/maps/observations.php (making weather and climate information useful), arena.tv/gastronomie/barfinder.jsp (finding a bar in Germany), cleancruising.com.au (Australian cruising site, another Map mashup), turnhere.com/sfbayarea (local Bay area guide), fishingmap.jp, bimrace.com (tracks your progress in race, shows elevation for a race and animates).
Anyway, what follows is notes/impressions from the different sessions I attended today.
The Opening Keynote was given by
Jeff Huber, Vice
Jeff explained the rationale behind conducting the event at 10 cities simultaneously that instead of bringing the developers to conference, take the conference to developers. There is approximately 1500 in Bay area, 5000 over all, 160 sessions over all.
housingmaps.com digged into the
are over 50,000 sites using these APIs all over the web.
Evolution of development over last several years: Start from open-source building blocks, Linux + Apache + MySQL + [Ruby+PHP+Java+Python]. Set of Web services as building blocks, for example Amazon S3 for storage.
Google's Developer Products are centered around three areas
- Integrate (Ability to take Google services to create your applications)
- Reach (How do I get distribution of my products, traffic to my site)
- Build (Building next gen web based applications)
Jeff talked about how Google provide infrastructure to build some of the products like [jogplanner, apartmentfinder, moviemaps. coolphotoeditor, syncingservice, certifiedusedcars, ok-i'm-kidding].google.com
Integrate (Google APIs + Mashup Editor)
GData APIs - Single RESTful Web service model, uses APP and RSS for reading/writing to Google's application. For example picnik (online photo editing), spanningsync (syncing mechanism to ical and isync on Mac), Honda Certified used cars (Google Base GData APIs, setting custom search).
Announced: Google Mashup Editor - Experimental product, online application to create mashups. When I went online to sign up, then I got the following message:
Google Mashup Editor
The Google Mashup Editor is currently in beta and access is limited to small number of developers during this testing period. If you would like to be notified when spots are available to participate in the beta please sign up here.
The demo created a mashup of CA state parks, feed from external
source, fed to Google Maps, and then link with Google Base for camping gear in 3
lines of code + HTML wrappers. Finally publish on googlemashups.com or
Reach (Gadget + Mapplets)
Google's network is the largest network, 500 million users per month. Gadgets API. Very open model, 100,000 gadgets have been created. Todo List Gadget had 6.7 million page views in last week, Expedia Fare Calendar Gadget (updates every time you are online with the latest roundtrip fares) had 4.7 million page views in last week.
10:35am: Seems like they turned on the Air Conditioner and it's
Announcement: Google Mapplets - Combined Gadgets and Maps API.
Orbitz mapplet, Weatherbug. The cool thing is you can show data from different sources such as Hotel and Weather on the same map. Combined Orbitz and WeatherBug with Google Maps - this combination is more useful than any of the individual sites. Called as "enabling a mashups of mashups".
Build (GWT + Gears)
Announcement: Google Gears - offline access for web apps, cross-browser, cross-platform, BSD-license. A way by which developers can add off line capability to your applications in a standard manner. Browser extension to allow web apps to work offline - Firefox, IE, Safari + Mac, Linux, Windows. Launching as open source project.
This is an evolutionary approach where end users continue using their application as they are using today w/o worrying about network connection, Developers continue building the application using their existing skills and code base, simply adapt to Gears API.
Also launching Google Reader Online - Working with Adobe, Opera and Firefox for industry collaboration. Kevin Lynch from Adobe showed a Flex/Apollo application with very similar online and offline experience.
Sergey Brin (co-founder of Google) made a cameo appearance. He talked about how the "person" involved in maintaining the Internet needs to be out-of-the-loop for it to truly self-sustaining and how Mapplets are leading towards that. And of course, payback to community which created such a fantastic ecosystem.
The complete presentation is available at doug.ricket.com/gdd2007. Doug Ricket, the speaker, used the preso from this link.
- onNYturf.com - Subway map of NY city on Google maps (classic API set - some data put on map, useful to users)
- trulia.com - Real estate search
- Panoramio - Upload your photos and tag them with a specific geographic location
- wikimapia.org - People are adding content all over the world to communicate (entirely user generated content)
The trend is towards more collaborative.
Load maps API package (indicated by "
file=api". There used to
be a limit to the number of times this API key can be used in a day. But this
limit is now removed. However if 5 million users are expected for the key, then
contact Google to ensure sufficient server capacity.
v=2" indicate that there is a big change in the API and
required changes in the application code. In reality, you'll get the latest 2.x
release, but behind the scenes versions are getting incremented and you get the
2.s is a more stable release for enterprise customers, solid and no bugs. You can also specify an exact version.
The fact that you can drag the map is the "big feature". Usability studies indicate that the cursor need to be changed to hand instead of mouse pointer (at least 2.51) - 2.50 you've pointer, 2.51 has a hand.
Because exact version numbers can be specified, almost infinite backward compatibility is available.
Key is registered for your specific website. For the public API, completely free but your site also need to be freely available. The basic API set has to be open to public. If you want to run it inside corporate firewall, then you need to use Enterprise and you get support from the Google team.
Follow along the presentation for all the detailed examples.
In Google Maps terms, InfoWindow is what is called as Balloon, Bubble or Pop-up. Marker is the red balloon, Title option to Marker is the tooltip.
Overlays is anything that can go over the map, for example Marker, Polylines, Polygons, Image and others. Can also create a custom overlay, basically any HTML can be converted into an overlay. There are few overlays that are available as default and then forums where people have shared their overlays.
Q. How far can you zoom in a particular level ?
A. Access to whatever is available to Google maps, in the area of 22 zoom levels. Certain level that is available all over the world. No API to get that level. The documentation says the default level available at all levels, but it's likely 18.
Q. When Maps are not available, can people provide their own tiles
Icons: Any icon (local or remote) can be used on the map. After 200 markers (on Firefox) and 100 on IE shows performance degradation.
Geocoder: Translating in address into lat/long and mapping it.
Full documentation: www.google.com/apis/maps/documentation
Q. Will Google add the ability to print satellite views of the maps
A. Not right now but working on it.
Q. Do you have Routing APIs
A. Come to the advanced session.
Q. Is there a way to highlight marker
Q. Can the Geocoder return the quality of geocode
A. It just returns null, check the documentation.
Q. Can you display a title for each marker without mouse over
A. Markers use an image, open source publicly create Overlay called TextMarker would serve that purpose, or create your own in 20-30 lines.
Q. Can Flash be embedded in Info windows
A. Put any HTML in the Info Window.
Q. Displaying Raster data on the map like a bitmap
A. Come to Advanced session. Take an image, put it down on the map anywhere and then add overlays. Or an API where you can specify tiles for different zoom levels.
"Go Forth and Create Mashups" was the key message.
The multi-cuisine (Asian, Mexican, Italian ...) lunch gave a feeling of what Googlers might be eating every day.
Jeff Dean, Google Fellow gave this talk explaining "Building a Computer System for the Worlds' Information". The talk was all about behind the scenes look at Google systems infrastructure.
Every Google service sees continuing growth in computational needs = More queries + More data + More results.
The goal is to create very large scale, high performance computing infrastructure => Hardware + Software systems to make it easy to build products. Focus on price/performance and ease of use.
The Hardware Design Philosophy is:
- Buy as many commodity class PCs (Prefer low-end server/PC-class designs)
- Larger number of unreliable computers than a big unreliable computer because reliability can be handled in software
- Rather parallelize and get latency down
- "Ultra-reliable" hardware makes programmers lazy
- Fault-tolerance to be built into software
The following set of pictures show how Google's infrastructure evolved over the years:
The three main components in Google's infrastructure are:
- Google File Systems (GFS)
- BigTable - Simple database doesn't support relational database but lets you spread data over multiple machines.
MapReduce - a programming model and library to simplify large-scale computations on large clusters.
Map phase extract relevant information from each record of the input, Reduce phase collects data together and produces final output.
- Fast (locality optimization, optimized sorter, lots of tuning work)
- Robust (handles machine failure, bad records)
- Easy to Use (little boilerplate, supports many formats)
- Scalable (can easily add more machines to handler more data or reduce the run-time)
- Widely applicable (can solve a broad range of problems)
Lots of products within Google use it.
BigTable - A large-scale storage system for semi-structured data
- Higher level API that a raw file system
- Somewhat like a database, but not as full-featured
- Useful for structured/semi-structured data
- Contents, crawl metadata, links, anchors, pagerank ...
- Per-user data
- User preference settings, recent queries/search results ...
- Geographic Data
- Physical entities, roads, satellite imagery ...
- Scales to large amount of data
Allows to do low-level storage optimization help performance significantly
- Distributed multi-level map
- Fault-tolerant, persistent
- Uses GFS underlying
- Thousands of servers
- Terabytes of in-memory data
- Petabytes of disk-based data
- Millions of reads/writes per second, efficient scans
- Servers can be added/removed dynamically, adjust to load imbalance
Lot of internal pre-processing such as satellite imagery. Largest BigTable manages ~3000TB of data spread over several thousand machines (larger cells planned)
Used by Google Print, My Search History, Orkut, Crawling/Indexing pipeline
Q. What flavors of Linux ?
A. Some using 2.4 and variants of 2.6
Q. Do you do destructive testing ?
A. Definitely yes. A battery of tests, take a data center offline, much more load than expected and see if it degrades gracefully and much more.
Q. Do you make an attempt to measure that interesting data grows than raw
A. Raw data requires computation to make interesting data. Not looked at different class of data.
Q. Will we see an open source version of GFS ?
A. There have been some discussions about it which would make difficult to open source it. There is an open source structure (Hudop?) that created a mimic of MapReduce.
Q. Do you modify the Linux kernel ?
A. Do modify as problems are discovered, lot of them are contributed back to the community.
Q. Few 1000 thousand servers, what policy is about recycling and green
A. Generally donate after outlived it's reasonable life and try to repair the machine instead of throwing it away.
First mashup - housingmaps.com. Paul did it before the Maps API.
Whole bunch of things before you can make/release the mashup.
Three parts to create a Mashup:
- User Interface
- Common set of Ajax UI elements (cross-browser compatible)
- Simple declaration of Ajax components (similar to using HTML tags)
- Flexible way of using UI controls programmatically
- Lots of machines and serving capacity
- Hosting or serve infrastructure with no setup up
- Simple data store modeled after a feed
- Ability to sandbox application for testing
- One click deployment of application
Experimental project built by a small team in NY
- An Ajax application that lets you built other Ajax applications
- Hosting container at googlemashups.com.
Feeds -> UI -> Test -> Publish -> Distribute
Mashups and Simple Applications made easy.
The speaker started with a survey and the results were:
- 80% use gadgets on desktop
- 50% use Google gadgets
- 5% develop gadgets
2007 - The Year of the Gadget ?
Specialized mini-application that runs in a web page or on the desktop. There are two types of widgets:
- Universal gadget
- Minimal web app
- No downloads necessary
- Needs online connection
- Desktop Gadget
- Runs on the desktop
- Advanced functionality
- Works online/offline
The talk focused on desktop gadgets.
Why develop desktop gadget ? Instantly reach millions of users, Easily distribute content, Increase traffic to your sites.
Gadget is a zip archive with .gg extension and the following structure:
An easier way to develop a gadget from scratch - Google Desktop Gadget Designer - allows you to easily experiment with WYSIWYG interface. It then showed a simple demo to create the gadget using Designer.
Advanced APIs - Add Flash, Graphics, Drag/Drop, Transparency, Download streams or Internet but following are the interesting ones:
- Communication API - send and receive data using the Google Talk framework. For example, multi-user games, social networking gadgets.
- Query API - find related information by searching for a specific term in the user index. For example, music or photo collection organizers. Searches the local desktop and prepare a list.
- Event API - Provide contextual information based on the users actions. For example, For web pages containing an address, show a map of the place.
- Desktop Google SDK
- Desktop Gadgets Developer Group
- FAQ / Knowledge Base
- Desktop Gadget Articles
- Google Desktop Blog
- Open source Desktop Gadgets
- Submit your gadget to Google to reach more users.
- The gadget will appear in the online gadget gallery and in the 'Add gadgets' panel on the users machine.
- Gadgets are automatically targeted and recommended to users who are most likely to use them.
- Populare gadgets will give you the chance to get into Google Desktop Hall of Fame
- 2007 - Year of the Gadget
- Desktop Gadget benefits:
- Easy to use, low resource requirements
- Powerful APIs offer opportunities for feature-rich designs
- Through the Google Desktop Gadget Gallery developers can:
- Reach millions of users around the world
- Target your gadget to users most likely to see it
- Distribute your content and increase traffic to your website
- Become famous as a top gadget developer
Why Custom Search ?
- Anyone, anywhere can create a substantially better search experience for their chosen topic
- No more complexity that creating a webpage
- The usual incentives apply: fame, fortune, pride ...
What Custom Search provides ?
- Harness the power of Google search
- Put search results on your website
- Customize the look & feel
- Invite friends
- Make money
Allows producing a custom search on demand.
That's the end of sessions. All in all, it was a great crash course for different Google offerings.