Saturday, May 15, 2010
Wednesday, May 12, 2010
Top 10 Largest Databases in the World
We all collected things as children. Rocks, baseball cards, Barbies, perhaps even bugs -- we all tried to gather up as much stuff as possible to compile the biggest most interesting collection possible. Some of you may have even been able to amass a collection of items numbering into the hundreds (or thousands). As the story always goes, we got older, our collections got smaller, and eventually our interests died out...until now. There are currently organizations around the world in the business of amassing collections of things, and their collections number into and above the trillions. In many cases these collections, or databases, consist of items we use every day. In this list, we cover the top 10 largest databases in the world:
Library of Congress
Not even the digital age can prevent the world's largest library from ending up on this list. The Library of Congress (LC) boasts more than 130 million items ranging from cook books to colonial newspapers to U.S. government proceedings. It is estimated that the text portion of the Library of Congress would comprise 20 terabytes of data. The LC expands at a rate of 10,000 items per day and takes up close to 530 miles of shelf space -- talk about a lengthy search for a book.
If you're researching a topic and cannot find the right information on the internet, the Library of Congress should be your destination of choice. For users researching U.S. history, around 5 million pieces from the LC's collection can be found online at American Memory.
Unfortunately for us, the Library of Congress has no plans of digitizing the entirety of its contents and limits the people who can check out materials to Supreme Court Justices, members of Congress, their respective staff, and a select few other government officials; however, anyone with a valid Reader Identification Card (the LC's library card) can access the collection.
By the Numbers
* 130 million items (books, photographs, maps, etc)
* 29 million books
* 10,000 new items added each day
* 530 miles of shelves
* 5 million digital documents
* 20 terabytes of text data
Central Intelligence Agency
The Central Intelligence Agency (CIA) is in the business of collecting and distributing information on people, places and things, so it should come as no surprise that they end up on this list. Although little is known about the overall size of the CIA's database, it is certain that the agency has amassed a great deal of information on both the public and private sectors via field work and digital intrusions.
Portions of the CIA database available to the public include the Freedom of Information Act (FOIA) Electronic Reading Room, The World Fact Book, and various other intelligence related publications. The FOIA library includes hundreds of thousands of official (and occasionally ultra-sensitive) U.S. government documents made available to the public electronically. The library grows at a rate of 100 articles per month and contains topics ranging from nuclear development in Pakistan to the type of beer available during the Korean War. The World Fact Book boasts general information on every country and territory in the world including maps, population numbers, military capabilities and more.
By the Numbers
* 100 FOIA items added each month
* Comprehensive statistics on more than 250 countries and entities
* Unknown number of classified information
Amazon
Amazon, the world's biggest retail store, maintains extensive records on its 59 million active customers including general personal information (phone number address, etc), receipts, wishlists, and virtually any sort of data the website can extract from its users while they are logged on. Amazon also keeps more than 250,000 full text books available online and allows users to comment and interact on virtually every page of the website, making Amazon one of the world's largest online communities.
This data coupled with millions of items in inventory Amazon sells each year -- and the millions of items in inventory Amazon associates sell -- makes for one very large database. Amazon's two largest databases combine for more than 42 terabytes of data, and that's only the beginning of things. If Amazon published the total number of databases they maintain and volume of data each database contained, the amount of data we know Amazon houses would increase substantially.
But still, you say 42 terabytes, that doesn't sound like so much. In relative terms, 42 terabytes of data would convert to 37 trillion forum posts.
By the Numbers
* 59 million active customers
* More than 42 terabytes of data
YouTube
After less than two years of operation YouTube has amassed the largest video library (and subsequently one of the largest databases) in the world. YouTube currently boasts a user base that watches more than 100 million clips per day accounting for more than 60% of all videos watched online.
In August of 2006, the Wall Street Journal projected YouTube's database to the sound of 45 terabytes of videos. While that figure doesn't sound terribly high relative to the amount of data available on the internet, YouTube has been experiencing a period of substantial growth (more than 65,000 new videos per day) since that figures publication, meaning that YouTube's database size has potentially more than doubled in the last 5 months.
Estimating the size of YouTube's database is particularly difficult due to the varying sizes and lengths of each video. However if one were truly ambitious (and a bit forgiving) we could project that the YouTube database will expect to grow as much as 20 terabytes of data in the next month.
Given: 65,000 videos per day X 30 days per month = 1,950,000 videos per month; 1 terabyte = 1,048,576 megabytes. If we assume that each video has a size of 1MB, YouTube would expect to grow 1.86 terabytes next month. Similarly, if we assume that each video has a size of 10MB, YouTube would expect to grow 18.6 terabytes next month.
By the Numbers
* 100 million videos watched per day
* 65,000 videos added each day
* 60% of all videos watched online
* At least 45 terabytes of videos
ChoicePoint
Imagine having to search through a phone book containing a billion pages for a phone number. When the employees at ChoicePoint want to know something about you, they have to do just that. If printed out, the ChoicePoint database would extend to the moon and back 77 times.
ChoicePoint is in the business of acquiring information about the American population -- addresses and phone numbers, driving records, criminal histories, etc., ChoicePoint has it all. For the most part, the data found in ChoicePoint's database is sold to the highest bidders, including the American government.
But how much does ChoicePoint really know? In 2002 ChoicePoint was able to help authorities solve a serial rapist case in Philadelphia and Fort Collins after producing a list of 6 potential suspects by data mining their DNA and personal records databases. In 2001 ChoicePoint was able to identify the remains of World Trade Center victims by matching DNA found in bone fragments to the information provided by victim's family members in conjunction to data found in their databases.
By the Numbers
* 250 terabytes of personal data
* Information on 250 million people
Sprint
Sprint is one of the world's largest telecommunication companies as it offers mobile services to more than 53 million subscribers, and prior to being sold in May of 2006, offered local and long distance land line packages.
Large telecommunication companies like Sprint are notorious for having immense databases to keep track of all of the calls taking place on their network. Sprint's database processes more than 365 million call detail records and operational measurements per day. The Sprint database is spread across 2.85 trillion database rows making it the database with the largest number of rows (data insertions if you will) in the world. At its peak, the database is subjected to more than 70,000 call detail record insertions per second.
By the Numbers
* 2.85 trillion database rows.
* 365 million call detail records processed per day
* At peak, 70,000 call detail record insertions per second
Although there is not much known about the true size of Google's database (Google keeps their information locked away in a vault that would put Fort Knox to shame), there is much known about the amount of and types of information Google collects.
On average, Google is subjected to 91 million searches per day, which accounts for close to 50% of all internet search activity. Google stores each and every search a user makes into its databases. After a years worth of searches, this figure amounts to more than 33 trillion database entries. Depending on the type of architecture of Google's databases, this figure could comprise hundreds of terabytes of information.
Google is also in the business of collecting information on its users. Google combines the queries users search for with information provided by the Google cookies stored on a user's computer to create virtual profiles.
To top it off, Google is currently experiencing record expansion rates by assimilating into various realms of the internet including digital media (Google Video, YouTube), advertising (Google Ads), email (GMail), and more. Essentially, the more Google expands, the more information their databases will be subjected to.
In terms of internet databases, Google is king.
By the Numbers
* 91 million searches per day
* accounts for 50% of all internet searches
* Virtual profiles of countless number of users
AT&T
Similar to Sprint, the United States' oldest telecommunications company AT&T maintains one of the world's largest databases. Architecturally speaking, the largest AT&T database is the cream of the crop as it boasts titles including the largest volume of data in one unique database (312 terabytes) and the second largest number of rows in a unique database (1.9 trillion), which comprises AT&T's extensive calling records.
The 1.9 trillion calling records include data on the number called, the time and duration of the call and various other billing categories. AT&T is so meticulous with their records that they've maintained calling data from decades ago -- long before the technology to store hundreds of terabytes of data ever became available. Chances are, if you're reading this have made a call via AT&T, the company still has all of your call's information.
By the Numbers
* 323 terabytes of information
* 1.9 trillion phone call records
National Energy Research Scientific Computing Center
The second largest database in the world belongs to the National Energy Research Scientific Computing Center (NERSC) in Oakland, California. NERSC is owned and operated by the Lawrence Berkeley National Laboratory and the U.S. Department of Energy. The database is privy to a host of information including atomic enegry research, high energy physics experiements, simulations of the early universe and more. Perhaps our best bet at traveling back in time is to fire up NERSC's supercomputers and observe the big bang.
The NERSC database encompasses 2.8 petabytes of information and is operated by more than 2,000 computational scientists. To put the size of NERSC into perspective, the total amount of spoken words in the history of humanity is estimated to be at 5 exabytes; in relative terms, the NERSC database is equivalent to 0.055% of the size of that figure.
Although that may not seem a lot at first glance, when you factor in that 6 billion humans around the globe speak more than 2,000 words a day, the sheer magnitude of that number becomes apparent.
By the Numbers
* 2.8 petabytes of data
* Operated by 2,000 computational scientists
World Data Centre for Climate
If you had a 35 million euro super computer lying around what would you use it for? The stock market? Building your own internet? Try extensive climate research -- if there's a machine out there that has the answer for global warming, this one might be it. Operated by the Max Planck Institute for Meteorology and German Climate Computing Centre, The World Data Centre for Climate (WDCC) is the largest database in the world.
The WDCC boasts 220 terabytes of data readily accessible on the web including information on climate research and anticipated climatic trends, as well as 110 terabytes (or 24,500 DVD's) worth of climate simulation data. To top it off, six petabytes worth of additional information are stored on magnetic tapes for easy access. How much data is six petabyte you ask? Try 3 times the amount of ALL the U.S. academic research libraries contents combined.
By the Numbers
* 220 terabytes of web data
* 6 petabytes of additional data
* Additional Databases
The following databases were unique (and massive) in their own right, and just fell short of the cut on our top 10 list.
Nielsen Media Research / Nielsen Net Ratings
Best known for its television audience size and composition rating abilities, the U.S. firm Nielsen Media Research is in the business of measuring mass-media audiences including television, radio, print media, and the internet. The database required to process such statistics as Google's daily internet searches is nothing short of massive.
United States Customs
The U.S. Customs database is unique in that it requires information on hundreds of thousands of people and objects entering and leaving the United States borders instantaneously. For this to be possible, the database was special programmed to process queries near instantaneously.
HPSS
There are various databases around the world using technology similar to that found in our countdown's second largest database NERSC. The technology is known as High Performance Storage System or HPSS. Several other massive HPSS databases include Lawrence Livermore National Laboratory, Sandia National Laboratories, Los Alamos National Laboratory, Commissariat a l'Energie Atomique Direction des Applications Militaires, and more.
Wednesday, March 31, 2010
News Thousands of Torrent Users Targeted by New Legal Action
Ref: http://www.maximumpc.com/article/news/thousands_torrent_userstargeted_new_legal_action
An organization known as the US Copyright Group has issued lawsuits against thousands of alleged movie pirates. The organization represents an alliance of independent film producers, with backing from the Independent Film & Television Alliance. The group is expected to file another round of lawsuits (possible as many as 30,000) in the coming weeks. The really troubling thing here is almost all of these are so called “John Doe” cases with IPs as the only identifying information. The group is trying to force ISPs to hand over names. Thus far, only one ISP has cooperated, resulting in 71 names. Using so called “pre-settlement” letters; the US Copyright Group has so far gotten five of those people to pony up some cash.
This scheme seems to be aimed at casting the widest possible net to increase the odds of scaring someone enough that they settle an automated lawsuit. This practice has been common in Europe for some time, but this is the first time it has reached American shores. Even the RIAA has abandoned suing individuals, but this new trend is growing. "We're creating a revenue stream and monetizing the equivalent of an alternative distribution channel,” said lawyer Jeffrey Weaver.
The US Copyright Group is using a new software technology that monitors large Torrent swarms and logs IP addresses. By moving ahead with these large numbers, they hope to reach a stable cost/benefit ratio. They really do see this as a revenue stream. Unfortunately, this means ISPs have a huge burden to be the middle men processing the complaints. Does anyone want to play devil’s advocate and defend this practice?Wednesday, March 24, 2010
Monday, February 15, 2010
Sunday, February 7, 2010
Friday, January 29, 2010
The 19 most complex and dangerous roads in the world
Col de Turini, France
Situated more than 1 mile above sea level, Col de Turini is a mountain pass situated in south of France in the Alps. It’s also part of a 20 miles rally stage of the Monte Carlo Rally of WRC, which combines 34 challenging hairpins and long stretches where cars top 111 mph. It is one of the most exciting roads on Earth. The pass was featured in the very first episode of Top Gear series 10, when the presenters went in search of the greatest driving road in the world. At its highest point, Col de Turini is 1607m high. In the north, the Col de Turini starts with a dazzling series of hairpins. Finally, we end up riding in a gorge, with a wild river on the left, and a steep rock-wall on the right.
2) Stelvio Pass, Italy
Located in the Eastern Alps in Italy, the Stelvio Pass Road connects the Valtellina with Merano and the upper Adige valley. This mountain road pass is situated at an altitude of around 1.7 miles above sea level. The road is particularly challenging to drive due to the presence of 48 hairpin bends, with the road becoming exceedingly narrow at some points, and some very steep inclines. With a height of 2757 meters, it is the highest paved mountain pass in the Eastern Alps and the second highest in the Alps, after the 2770 m high Col de l’Iseran. While it might not be as dangerous as the other routes, it is certainly breathtaking. The toughest and most spectacular drives are from the Prato side. The mountain pass is one of the best continuous hairpin routes in the world.
3) Leh–Manali Highway, India
Photo credit 1
The Leh-Manali Highway is situated in India and spans over a length of 297 miles among the Himalaya mountain range. It passes through some of the worlds highest mountain passes in the world, with a mean altitude in between 2 to 3 miles above sea level. The road is one of the most complicated and challenging roads in the world, with snow, landslides and terrain making the journey exceedingly difficult for anything other than a capable four wheel drive vehicle. The road was built and is maintained by the Indian Army.
4) The Puxi Viaduct, Shanghai
Photo credit 1
This is one of Shanghai’s busiest and largest interchange that caters to thousands of vehicles every hour. It has five levels of bridges that help connect two of the cities busiest highways, directing vehicles without much fuss.
5) The Judge Harry Pregerson Interchange, LA
Photo credit: 1
The Judge Harry Pregerson Interchange is situated in Los Angeles, CA and is one of the most complicated interchanges in the country. It permits entry and exit in all directions between the I-105 and the I-110. It’s a stack interchange with layers of bridges making a complicated network of roads allowing smooth flow of traffic though both the interstate highways. This interchange was opened in 1993. It is a 4 level interchange with a restricted access lane that can be used by high-occupancy vehicles.
6) The Road of death, Bolivia
The North Yungas Road (also known as the El Camino de la Muerte, ‘Road of Death’ in Spanish) is a 43 mile road connecting La Paz and Coroico, 35 miles northeast of La Paz in Bolivia. Famous for its extreme danger, it was christened as the “world’s most dangerous road” in 1995 by the Inter-American Development Bank. The single-lane width, extreme drop offs, and lack of guardrails, only add to the danger lurking behind. Further, the fog and rain can make visibility poor and the road surface muddy, loosening rocks from the hillsides above. It is estimated that 200 to 300 travelers are killed per year on this treacherous road. Although, the old North Yungas Road is much less used by traffic nowadays, an increasing number of adventure bikers travel it for the thrills.
7) Russia’s Lena Highway, the Highway from Hell
The last 600 miles of the Russian Federal Highway from Moscow city to the Siberian city of Yakutsk is called the “Lena Highway”. This bizarre road runs parallel to the River Lena on the final leg to Yakutsk. As if the road of mud was not a big problem, Yakutsk is considered one of the the coldest cities on earth, with January temperatures averaging -45 °F. But surprisingly, it is only in the summertime that the road becomes impassable. Whenever it rains in summer, the road virtually becomes a slush pit making it impossible for the vehicles to pass through it. This being the only road to Yakutsk makes the traffic heavy and even more complicated to negotiate.
8. Gravelly Hill Interchange, Burmingham, UK
Photo credit: 1
Gravelly Hill Interchange, nicknamed ‘the Spaghetti Junction’, is the 6th junction of the M6 motorway, where it joins the A38 Aston Expressway in Birmingham, UK. The name “Spaghetti Junction” was coined by Roy Smith, a journalist from the Birmingham Evening Mail in the 1970s. The areal view of the junction sure tells us why it is called the Spaghetti Junction. Spanning an impressive 30 acres, the junction serves 18 routes and includes 4 km of slip roads. Across 6 different levels, there are 559 concrete columns, reaching up to 24.4 m in height. The engineers had to elevate 13.5 miles of the motorway to accommodate 2 railway lines, 3 canals, and 2 rivers. It’s the most complicated junction in United Kingdom.
9) Russian-Georgian “Military” Mountain Roads
Photo credit 1
When they are not covered in sheets of snow, then it’s the thick, grueling mud. These remote highways would probably swallow your car in the snow or mud. Though neither affect the locals who drive their Lada cars down it regularly. Situated in the Caucasus mountains, these roads are to be tackled only by the Russian military which probably explain why they lack any official designation. The harsh surface, along with the problems posed by snow, makes this road almost inaccessible during winter. The seldom used road connects Russia and Georgia and assumes of strategic importance for both countries.
10) Guoliang Tunnel Road, China
The magnificent tunnel road in the Taihang mountains was built by 13 local villagers headed by their chief, Shen Mingxin, and took around five years to finish. Many villagers lost their lives in accidents during construction of the tunnel but the others continued relentlessly. The tunnel was opened to traffic on May 1st, 1977. The 1200 meter long tunnel is about 5 meters high and 4 meters wide. It is located in the Henan Province of China. The Guoliang tunnel is another addition to most dangerous and complicated roads to travel. Dubbed as “the road that does not tolerate any mistakes”, most accidents in the tunnel are primarily caused by the neglect of the traveler. Nonetheless, it is an extremely scenic route and is a key destination on the Chinese tourism map.
11) Taroko Gorge Road in Taiwan (Chungheng)
Photo credit: 1
The Taroko Gorge Road in Taiwan is another mountain route in China made by carving out rocks, like the Guoliang Tunnel road. The road passes through the Taroko national park alongside the Taroko Gorge. The road is an appeal to the tourist, as well as a mode of transportation of marble found abundantly in the Gorge.
12) Pasubio (Vicenza), Northern Italy
Photo credit: 1
This is a hiking trail made out of an ancient road trail. The road serves mostly for motorcycles and certain types of car. The road is dangerously narrow and slippery, spanning many cliff faces and tunnels with stunning scenery, making this a popular destination for adventurous travelers.
13) The Halsema Highway in the Philippines
photo credits 1
The Halsema Highway runs through the Central Cordillera Valley in Philippines. It is also called the Baguio-Bontoc Road. The road is approximately 150 miles long and is mostly unpaved. The road runs through steep cliff faces which barely have any guard rails or other safety devices installed. The narrow roads and steep cliff faces make the road almost impassable during the rainy season. It’s known for the rock slides and mud slides and buses driving dangerously fast on its narrow passage. There are plenty of accidents and many overturned buses on a yearly basis. There are sheer drop offs of more than 1000 feet without a safety guard rail. This route is for sure one of the most dangerous roads in the world.
14) Trollstigen in Norway
The Fjord in Norway has many roads that attract tourists. The most notable among them is the Trollstigen which is a series of stunning roads with a breathtaking view of a few waterfalls. The word Trollstigen means the Troll Ladder. The road, though not lacking in safety standards, takes a lot of concentration and driving skill to conquer. The vertigo-inducing steep inclines, intense set of hairpins and narrow roads leave no margin for error. However, once you are at the top, the view is just breathtaking. The narrow road leaves us with extremely few possibilities for vehicles to pass each other. The frequent rockfalls in the region have resulted in some upgrades to the road in 2005. At the top, there is a viewing balcony which overlooks the road and the Stigfossen waterfall, a 320 m long waterfall which falls down the mountain side.
15) Los Caracoles Pass in Andes
Photo credit: 1
This road passes though the Andreas Mountains on the way between Chile and Argentina. Los Caracoles is a series of hard switchbacks on an extremely steep incline. The road has many steep inclines and hairpins without any safety guard rails. The road is covered with snow for the most part of the year. The snow together with nature of the road requires extreme patience and skill to negotiate. However, this road is maintained pretty regularly and does not have a morbid accident record. Cargo trucks and even double-Decker tourist buses travel through the road on a daily basis, and it’s quite an experience.
16) Iroha-zaka winding road, Japan
Iroha-zaka winding road is the main route that connects central Nikko and Oku-Nikko. The First Iroha-zaka is used to come down, and the Second Iroha-zaka to go up. Each corner has an ancient Japanese alphabet, and you will see it in alphabetical order starting from I-ro-ha and hence the name. The road was used by ascetics in the past. The number of curves on the road was 48, matching the 48 letters of the ancient Japanese alphabet. Therefore, the tourist guides started to call the slope Iroha-zaka. After the construction of the second Iroha-zaka there were 50 curves, but 2 were decreased to remain corresponding with the 48 letters. How’s that for complicated?
17) Van Zyl’s Pass, Namibia
Photo credits: 1
Van Zyl’s Pass, or the DR3703, located in Namibia, is a classic extreme road. It is not exactly a road, just a route made over the mountain by the travelers over time. The outrageously steep pass provides a pure adrenaline rush, but the route that leads up to it is a 10-15km of tough driving where one has to dodge their way through rocks, boulders, badlands and ravines. At the end, the road descends to the ancient glacial valley called Marienfluss valley, which is one of the planet’s most beautiful sights that await only the brave-hearted.
18) El Espinazo Del diablo, Mexico
El Espinazo Del diablo or ‘The devil’s backbone’ is the mountain pass in Durango, Mexico. It’s about 5 hours long, and it was the only road from Durango to Mazatlan Sinaloa for a long time. We have heard many cautionary tales about crossing the devil’s backbone, El Espinoza Del Diablo, But the road is exceptionally well maintained and there are many cautionary signs marking most of the hazards. Of course these are in Spanish, so keep an electronic translator or a dictionary, handy. Pull out spots are frequent, so you can easily stop anytime you want. There are some tight curves, too. So tight that a truck needs all of the road to make it around. These hinder potential two way traffic in these regions. However, stunning rock formations rising around you and the lush, green vistas stretching on for impossible distances make every inch of the drive breathtaking.
19) Lysebotn Road, Norway
Photo credit: 1
This is probably the most fun road you can travel on four wheels, and then maybe on your two legs checking out the various hiking trails leading from the area. In fact, this might be considered the most breathtaking place in Europe. It all starts with the narrow road up the steep walls of the Lysefjord, Norway. It has 27 switchbacks and a 1.1 km long tunnel at the bottom, with 3 switchbacks inside. The last 30 km of Lysebotn road is a true roller-coaster! It’s narrow but has a perfect surface, winding left and right all the time. If you happen to ride a motorcycle in Norway, then this is the road you simply cannot afford to miss!