#|
With the entire might of the Gotham City Police Department and Gotham's rich and powerful coming down on his head, Batman must find this imposter and somehow clear his name…but how can you prove your innocence from behind a mask? This slipcase box set includes softcover editions of Shazam! All the damage from all the Crises was undone, and heroes long thought gone returned from whatever exile they had been in. Most of them, at least.
Alan Scott, the Green Lantern of the Justice Society of America, has noticed some of his allies are still missing in action, and he's determined to find them. There are others, though, who would rather remain hidden than explain themselves, like Roy Harper, a. Arsenal, a man who should be dead but now is not. Plus, what does all this mean for the DCU's place in the Multiverse?
On opposite sides of a dimensional divide, both Barry Allen and President Superman ponder this question. Not to mention the Darkseid of it all! Or a team of Multiversal heroes called Justice Incarnate! Some met him in childhood; some met him months ago. And Walter's always been a little…off. But after the hardest year of their lives, nobody was going to turn down Walter's invitation to an astonishingly beautiful house in the woods, overlooking an enormous sylvan lake.
It's beautiful, it's opulent, it's private—so a week of putting up with Walter's weird little schemes and nicknames in exchange for the vacation of a lifetime? Why not? All of them were at that moment in their lives when they could feel themselves pulling away from their other friends; wouldn't a chance to reconnect be…nice? Don't miss the first collected edition of 's smash-hit horror sensation—so you can be all caught up when The Nice House on the Lake returns with issue 7!
But she doesn't have time to worry about the past…she has to focus on finding a way to get rid of Trigon for good. Garfield Logan still can't believe he has the power to transform into animals. But controlling his newfound abilities is difficult, and their unpredictable nature could have dangerous consequences. Knowing his parents kept this secret hidden from him only makes Gar feel more alone. The heroes are both seeking answers from the one person who seems to have them all figured out: Slade Wilson.
When their paths cross in Nashville, Raven and Gar can't help but feel a connection, despite the secrets they try to hide from each other. It will take a lot of trust and courage to overcome the wounds of their pasts.
But can they find acceptance for the darkest parts of themselves? Or maybe even love? These unique origin stories define what it means to have the strength to face and accept who you really are. In Teen Titans: Raven , when year-old Raven Roth loses her memories and has to start life over in New Orleans, strange things begin to happen. What if she's better off not knowing her past? Then new powers and elevated status give Gar everything he's hoped for—but at what price? It's the meet-cute that turns into the adventure of a lifetime, as they discover that trusting a new friend can be everything, and trusting the wrong adult can be a disaster.
Is it possible the Metropolis Marvel could be losing a step? The Man of Steel's struggles in taking down the creatures from the Breach would suggest as much! If he's going to continue to protect the people of Earth, he'll have to adapt—especially with threats like Mongul out there waiting to launch their biggest attacks on the Earth yet. After a war-torn battleship escapes Warworld and makes the perilous journey to Earth, Superman searches for answers about the identities of its mysterious refugees and their apparent link to the planet Krypton.
Could there be other Kryptonians in the universe? Meanwhile, Atlantean scientists study the wreckage of the Warworld vessel…and make a shocking discovery that could change the balance of power on Earth. After the heart-stopping events of Action Comics , Superman and the surviving members of the Authority see a side of Warworld they never knew existed. In the lower catacombs, Superman finds another survivor of the lost Phaelosian race of Krypton, a scientist turned enslaved gladiator with much to teach Superman of his new home, including how to survive…and maybe, in time, how to escape.
Meanwhile, Superman's quest to turn the hordes of Warworld against their masters begins. Jackson Hyde's made some daring escapes in his time on the run, but there's no avoiding the reunions that his underwater motherland has in store for him. Both surprise family time and a long-awaited romantic interlude leave Jackson questioning his life on the surface.
And with all the problems Jackson left behind in Atlantis, it's getting harder not to ask himself—is Xebel where he really belongs? After last issue's revelation, Aquaman and Green Arrow must double their efforts to escape and thwart Scorpio's plot to rewrite time to their own agenda! With their new secret muscle car and their new secret patrol route, Batgirls Cassandra Cain and Stephanie Brown find moving to their new neighborhood—thanks to Oracle instructing them to "lay low"—that much easier to bear because they have each other.
Steph begins witnessing some strange actions through the window of the building across the street and can't help but investigate if the recent murders are connected to them! Meanwhile, Oracle realizes the most effective way for the girls to wear her newly upgraded comms is by piercing their ears, and Cass freaks out! After a disastrous first day on their new jobs as hero business owners and operators, the duo has found themselves kidnapped and stranded on an alien world.
Who's to blame? Well, the Omnizon of course! Welcome to her home planet, Br'Honn, where a battle to the death between friends is just another Tuesday! DC VS. It's hero versus hero in this blood-drenched chapter…with clues to who the new Vampire King might be!
To escape the horror, the Deathstroke Inc. Now John is the only one who can stop the Lightbringer's plans, but in order to do so, he must choose a new path forward, one that will change his role in the DC Universe forever!
His reputation is in tatters, his suit nearly destroyed, and his body beaten, and still nothing can stop Curtis from getting his revenge. It's time for Edwin Alva to realize just how big of a mistake he made in underestimating the power of Hardware's rage! One track leads straight to Gotham Central Station, where hundreds of lives are at risk, but the other track…that one leads to my best friend and sidekick, Kevin.
Sacrifice the one to save the many? I hate that Philosophy crap, and I'm really starting to hate trains. His murderer couldn't be Blue Beetle…could it? What in the Multiverse could the Royal Flush Gang be after? How does it connect to Black Adam's trial? Find out here! To save her, President Superman, Flashpoint Batman, and the rest of the Justice League Incarnate team up with Earth heroes Spore and Nimrod Squad; meanwhile, a villain from Multiversity returns to stake their claim on the crack in the Multiverse and the power that lies beyond.
Daffy is sure their two-on-one advantage will bring them fame and fortune…until he discovers that their opponent is none other than the heavyweight champion, the Crusher…whose previous opponents each left the ring on a stretcher! America's longest-running humor magazine continues to skewer everything pop culture! This pet-themed issue features a wide variety of classic parodies plus vintage MAD favorites like Spy vs.
MAD will surely make the whole family howl, meow, chirp, whinny, moo, and cock-a-doodle-doo with chuckles. Go fetch your copy today!
Nightwing gets an updated suit starting this issue! Meanwhile, after the distressing events of rescuing Haley from getting dognapped by bad guys, Nightwing discovers there are way more hits on Dick Grayson than he realized, thanks to going public about his fortune, and he needs to find a clever way to be Dick Grayson and Nightwing at the same time.
Concerned about her sister, Nubia leads the charge. What evil from Tartarus has found its way into the very soul of Doom's Doorway's latest champion? To find out, our queen will have to delve deep into her haunted past for clues. Can she heal herself from old wounds in time to save another from making the same mistakes? Find out in another exciting chapter of Nubia's solo adventures! It's not easy being Red Tornado! As the boss, he's got everyone relying on him for their next paycheck.
Enter Minute Man, a s has-been superhero looking for 15 more minutes of fame…or at least a way to pay for Miraclo pills. Without them, he's a super-zero and he's willing to do anything for one more chance at power. After that go back to the Home screen of your Android Box by pressing the Home Button on your remote.
My problem is in the title, i can't download any. If you own a separate Android device, using apps2fire to install content to your Powerful Downloader for Android: - downloading from internet up to three files simultaneously; - accelerated downloading by using multithreading 9 parts With internal app sharing, you can quickly share an app bundle or APK with addresses or make your app available to anyone you share a download link with.
They're automatically re-signed with an Internal App Sharing key, which is After you upload an APK to the internal app sharing upload pagefor the first time, the. You can make simple modifications to Dropbox links to share files the way you want. Download and Convert YouTube videos for free with Viddly. The emulator will automatically pick up the APK file and install the same on your. There is no modification in the APK file, you can trust us on. So, the users who need to Kodi We then pretty-print this data:.
As you can see, we got some information back rather quickly. There's the icon that was used, a preview of the text, the title, even the language, date and HTML have been returned.
You'll notice there's no author, however. Let's change this and request some more values. Add this line to the query array:. Refreshing the screen now gives us this:. But, the source code of the article notes several other tags:.
Why is the result so very different? It's precisely due to the reason we mentioned at the end of the very first paragraph of this post: what we humans see takes precedence. Diffbot is a visual learning robot, and as such its AI deducts the tags from the actual rendered content - what it can see - rather than from looking at the source code which is far too easily spiced up for SEO purposes.
Is there a way to get the tags from the source code, though, if one really needs them? Furthermore, can we make Diffbot recognize the author on SitePoint articles? With the Custom API. The Custom API is a feature which allows you to not only tweak existing Diffbot API to your liking by adding new fields and rules for content extraction, but also allows you to create completely new APIs accessed via a dedicated URL, too for custom content processing.
Go to the dev dashboard and log in with your token. Then, go into ""Custom API"". Your screen should look something like this:. You'll immediately notice the Author field is empty. You can tweak the author-searching rule by clicking Edit next to it, and finding the Author element in the live preview window that opens, then click on it to get the desired result.
However, due to some, well, less than perfect CSS on SitePoint's end, it's very difficult to provide Diffbot's API with a consistent path to the author name, especially by clicking on elements. Instead, add the following rule manually:. You'll notice the Preview window now correctly populates the Author field:. In fact, this new rule is automatically applied to all SitePoint links for your token. If you try to preview another SitePoint article, like this one, you'll notice Peter Nijssen is successfully extracted:.
Ok, let's modify the API further. We need the article:tag values that are visible in source code. Doing this requires a two-step process. A collection is exactly what it sounds like - a collection of values grabbed via a specific ruleset. This means ""find all meta elements in the HTML that have the property attribute with the value article:tag"". Collection fields are individual entries in a collection - in our case, the various tags.
Click on ""Add a custom field to this collection"", and add the following values:. Click Save. You'll immediately have access to the list of Tags in the result window:. Change the final output of the diffbotDemo action to this:.
Here's the output the above line of code produces:. Diffbot is a powerful data extractor for the web - whether you need to consolidate many sites into a single search index without combining their back-ends, want to build a news aggregator, have an idea for a URL preview web component, or want to regularly harvest the contents of competitors' public pricing lists, Diffbot can help.
With dead simple API calls and highly structured responses, you'll be up and running in next to no time. We'll also host the library on Packagist, so you can easily install it with Composer.
The main function of Diffbot is to turn the open web into an easily digestible API. So for example, AOL is one of Diffbot's clients. It has numerous online media properties using different content management systems. Rather than trying to figure out how to organize all of that, it uses Diffbot to read new articles from all of those properties and extract that data into one simple API.
That way its new tablet magazine app, Editions, can pull stories from across all AOL properties and display them on the iPad in real time.
So far Diffbot has been limited to understanding front pages and article pages, but founder Michael Tung told us that the company is now expanding to cover a much wider range. Right now the company has two basic services. It can scan URLs that a customer sends them or it can monitor a URL for a customer and alert them to changes, something Tung says many clients are using to keep an eye on their competition.
The company works on a freemium model, with the first 10, API calls per month being free and tiered pricing after that. The new funds, says Tung, will be used to scale the companies servers to keep up with demand and hire machine learning experts in a crowded and competitive marketplace.
Diffbot guarantees a high uptime, but failures sometimes do happen — especially in the most resource intensive API of the bunch: Crawlbot. Not by a lot, but enough to be noticeable in the API Health screen — the screen you can check to see if an API is up and running or currently unavailable if your calls run into issues or return error Diffbot is a visual learning robot, and as such its AI deducts the tags from the actual rendered content — what it can see — rather than from looking at the source code which is far too easily spiced up for SEO purposes.
A collection is exactly what it sounds like — a collection of values grabbed via a specific ruleset. Collection fields are individual entries in a collection — in our case, the various tags. Is there any open-source web scraping tool such as Scrapinghub or DiffBot? Kimono even hosts your data for you and lets you update whenever you tell it to. This is perfect if you have one site with a list of hundreds of links to blog articles, and you want the content inside each article.
Kimono lets you scrape the list and use it as a source of links for another API to crawl. Fox vs. CNN: Who's got Obama on the mind? I should tell you that I work for Kimono, but. I truly think the product is awesome. Additionally, since then, the design of the pages we processed has changed, and thus the API no longer reliably works. The following few commands will bootstrap the Vagrant box, create the project folder, and install the Diffbot client. If we now give index. This is all we need to init Diffbot.
First, we need to rebuild our API from the last post, so that it can become operational again. After entering a sample URL like www. We also need to define a collection which would gather all the article cards and process them. Making a collection entails selecting an element the selector of which is repeated multiple times. Besides title and primary category, we should also to extract the date of publication, primary category URL, article URLs, number of likes, etc.
If we now access our endpoint directly rather than in the API toolkit, we should get the fully merged 9 pages of posts back, processed just the way we want them.
We can see that the API successfully found all the pages in the set and returned even the oldest of posts. This step is, in a way, optional. We need two new classes: an Entity Factory, and an Entity.
As per the docs, we have an abstract class we can extend. We extend the abstract entity and give our new entity its own namespace. This is optional, but useful. At this point, the entity would already be usable — it is essentially identical to the Wildcard entity which uses magic methods to resolve requests for various properties of the returned data which is why the getBio method in the example above worked without us having to define anything. But the goal is to have the AuthorFolio class verbose, with support for custom, SitePoint-specific data and maybe some shortcut methods.
We can also tell PHPStorm that the class will have an articles property using the property tag, so it stops complaining about accessing the field with magic methods:. Other methods we could define are totalLikes, activeSince, favoredCategory, etc.
We merged the original API-to-entity list with our own custom binding, thereby telling the Factory class to both keep an eye on the standard types and APIs, and our new ones. This means we can keep using this factory for default Diffbot APIs as well. To make our classes autoloadable, we should probably add them to composer. We activate these new autoload mappings by running composer dump-autoload.
Next, we instantiate the new factory, plug it into our Diffbot instance, and test the API:. In this tutorial, by using the official Diffbot client, we constructed custom entities and built a custom API which returns them. We saw how easy it is to leverage machine learning and optical content processing for grabbing arbitrary data from websites of any type, and we saw how heavily customizable the Diffbot client is.
In this post, we'll extract the links to the author's social networks. If you look at the social network icons inside an author's bio frame on their profile page, you'll notice they vary. There can be none, or there can be eight, or anything in between.
What's worse, the links aren't classed in any semantically meaningful way - they're just links with an icon and a href attribute. This makes turning them into an extractable pattern difficult, and yet that's exactly what we'll be doing here because hey, who doesn't love a challenge?
To get set up, please read and go through the first part. When you're done, re-enter the dev dashboard. The logical approach would be to define a new collection just like for posts, but one that targets the social network links. Then, just target the href attribute on each and we're set, right?
As you can see, we get all the social links. But we get them all X times, where X is the number of pages in an author's profile. This happens because the Diffbot API concatenates the HTML of all the pages into a single big one, and our collection rule finds several sets of these social network icon-links.
Intuition might lead you to use a :first-child pseudo element on the parent of the collection on the first page, but the API doesn't work like that. The HTML contents of the individual pages are concatenated, yes, but the rules are executed on them first. In reality, only the result is being concatenated. This is why it isn't possible to use main:first-child to target the first page only.
Likewise, at this moment the Diffbot API does not have any :first-page custom pseudo elements, but them appearing at a later stage is not out of the question. How, then, do we do this? Diffbot allows you to define several custom rulesets for the same API endpoint, differing by domain regex. When an API endpoint is called, all the rulesets that match the URL are executed, the results are concatenated, and you get a unique set back, as if it was all in a single API.
This is what we're going to do, too. Enter the same name as the one in the first part in my case, AuthorFolio. Then, change the domain regex to this:. This tells the API to only target the first page of any author profile - it ignores pagination completely. Next, define a new collection. Call it ""social"" and give it a custom field with the selector of.
Name the field ""link"", and give it a selector of ""a"" with an attribute filter of href. Save, wait for the reload, and notice that you now have the four links extracted:. But having just the links there kind of sucks, doesn't it? It would be nice if we had a social network name, too. SitePoint's design, however, doesn't class them in any semantically meaningful way, so there's no easy way to get the network name. How can we tackle this? Custom fields have three available filters:. We'll be using the third one - read more about them here.
Add a new field to our ""social"" collection. Give it the name ""network"", the selector a, and an attribute filter of href so it extracts the link just like the ""link"" field.
Then, add a new ""replace"" filter. Luckily, each of those has pretty straightforward URLs with full domain names, so regexing the names out is a piece of cake. Save, wait for the reload, and notice that you now have a well formed collection of an author's social links. Finally, let's fetch all this data at once. According to what we said above, executing a call to an author page with the AuthorFolio API should now give us a single JSON response containing the sum of everything we've defined so far, including the fields from the first post.
Let's see if that's true. Visit the following link in your browser:. As you can see, we successfully merged the two APIs and got back a single result of everything we wanted. We can now consume this API URL at will from any third party application, and pull in the portfolio of an author, easily grouping by date, detecting changes in the bio, registering newly added social networks, and much more.
In this post we looked at some trickier aspects of visual crawling with Diffbot like repeated collections and duplicate APIs on custom domain regexes. We built an endpoint that allows us to extract valuable information from an author's profile, and we learned how to apply this knowledge to any similar situation.
Did you crawl something interesting using these techniques? Did you run into any trouble? Let us know in the comments below! In the previous post on Analyzing SitePoint Authors' Profiles with Diffbot we built a Custom API that automatically paginates an author's list of work and extracts his name, bio and a list of posts with basic data URL, title and date stamp.
What is the algorithm used by Diffbot for extracting web data? What this means: when analyzing a web document, our system renders the page f Companies like Google, Facebook, and Baidu — which are all working on artificial intelligence — have the benefit of massive amounts of data at their fingertips that they and their data entry employees can use to categorize and define the web in a language that AI software can later feed into their algorithms.
The major expenses for Diffbot had been electricity and bandwidth, Tung says. Unlike other artificial intelligence deep learning projects that rely on humans to classify web pages, Diffbot uses only the proprietary algorithms that it created itself and has refined over the years, according to Tung.
If artificial intelligence is to achieve the promise and potential peril inherent in the technology, it still needs to be taught. Tung compares it to teaching a child.
Research into artificial intelligence, and the ability to develop sentience in machines, sits at the intersection of a few very large trends in computing. It combines the development of new, and newly powerful, chipsets that can process complex increasingly quickly; the development of new kinds of database software that can organize massive amounts of data more flexibly, and the development of a nearly ubiquitous arrays of sensors and systems to collect that data. Tung calls it the Manhattan project for AI — except computers are the researchers developing the bomb.
Diffbot was always going to make money. He lived on a diet of beans and rice and ramen, alternating working on the math at the core of the software with filing patent applications for money. With the initial seed money from StartX, Diffbot was able to continue its research and launch its first, revenue generating, products.
For every hit to our server we earn. In retrospect it was a decision that Tung was happiest about. Many of those on-demand customers are still on board. It was time. So the company started spidering the web to speed up its data collection. Lofty goals attract big investors, and Diffbot has attracted some of the biggest.
And Felicis Ventures, which is building a sizable portfolio of artificial intelligence companies. A coterie of new angels and other institutions joined as well — all of them also bold-faced names in the Valley. And hitting profitability last year as one of the first AI startups to do so was a turning point. Has anybody ever used Diffbot for a web scraping solution?
Typical examples include a product page on Amazon. Core categories include people skills, employment history, education, social profile , companies, locations mapping data, addresses, business types, zoning information , articles every news article, dateline, byline from anywhere on the web, in any language , discussions chats, social sharing, and conversations , and images organized using image recognition and metadata collection.
Among those clients are Microsoft, eBay, Yandex, and DuckDuckGo, which are using it to enhance the quality of their search results. In a demo, Tung showed me how it worked. Say you wanted to perform a one-off search for a brand of shoe. Looking for news articles instead? Searching for a person, on the other hand, pulls up a CV-like work history pieced together from dozens or hundreds of bios, articles, and publicly available profiles. Diffbot launched in and counts 28 employees among its core staff of engineers and data scientists.
Diffbot brings big-time search poobah aboard to help it scale,dT,Barb Darrow,Gigaom,"Diffbot,web search engine","Diffbot is a small company with a big plan: to convert gazillions of web pages into machine-readable format that can be used and reused by lots of applications.
The new hire? Matt Wells, who created Gigablast, a pioneering search engine. Some background: Gigablast was one of the first search engines to do real-time web indexing. At one point, back in the in the mid s, its index hit the 12 billion page mark, second only to Google s goog but while Google was built with lots of people and resources, Gigablast was essentially a one-man show.
Matrix Partners also participated in the round. Last August, the company publicly debuted its first APIs, which allow developers to build apps that can automatically extract meaning from web pages. For example, the Front Page API is able to analyze site homepages, and understands the difference between article text, headlines, bylines, ads, etc. The Article API can then extract clean article text, images and videos.
Today, Diffbot has categorized the web into about 20 different page types, including homepages and article pages, which are the first two types it can now identity. Going forward, Diffbot plans train its bots to recognize all the other types of pages, including product pages, social networking profiles, recipe pages, review pages, and more.
Today, Diffbot is releasing its first set of APIs, now open to all developers for free. The launch has the potential to dramatically impact the types of applications developers can build, and for consumers, it means a whole host of intelligent applications are about to emerge.
The Article API extracts clean article text, pictures and tags. For example, see Readably. Follow API: This is used to track the changes or updates made to any webpage.
Diffbot automatically determines the part of the page that the developer wants to follow and extracts metadata like title, images, text summary and more, then segments the page into meaningful sections See above photo. Nuance uses the technology to improve its natural language processing in a product for doctors, which requires comprehension of complex medical terminology.
SocMetrics sends bit. These are just a few big-name examples. There are smaller, but just as innovative use cases out there, too. The new self-serve platform for developers is free up to 50, API calls per month. The Managed plan for Enterprise requires custom pricing. This tells the API to only target the first page of any author profile — it ignores pagination completely.
Most web pages fall into a handful of broad categories — news articles, front page, images, events and extracts. Customers include Instapaper which can take that structured data and repurpose it for use on mobile devices, he said. Academics and big vendors including Google s goog , Microsoft s msft and Yahoo s yhoo are all working to better understand web pages. Google Research and Microsoft Research are no doubt doing similar work, the difference being they are keeping it as a black box,Tung said.
It sounds a little geeky, but it provides a way for developers and publishers to analyze the web and organize it in ways that are very easy to digest for people, especially mobile users. The company, which has just opened its first API to the public, is encouraging developers to start using it for a variety of applications. What might these apps look like? Well, AOL s aol joined a private beta earlier this year and is now using Diffbot to help personalize its Editions news reader iPad app, s aapl by grabbing content and pulling out the top news stories from other sources.
Diffbot is able to look at a content source and determine what kind of page it is, what the elements are such as headlines, images, advertisements and contextually understand the content on the page. It can determine what the top story is on a news site. These companies are often doing this kind of work themselves but will now have an option in Diffbot.
Nuance uses the Diffbot API to build large domain-specific text corpuses to train its natural language processing system to recognize speech more accurately in specific areas like medicine.
Hacker News Radio uses Diffbot to take the top Hacker News stories and turn them into text-to-speech content for an online radio station. Tung said Diffbot can really be useful in mobile applications, which have limited screen real estate and can use extra intelligence to help present content. A lot of the apps are using us because that want to incorporate web data and display it in a better way on mobile devices or do something custom. The Frontpage API analyzes home pages and indexes things like headlines, bylines, images, articles and ads, while the Article API can extract article text, pictures, and tags from news pages.
A second main API called Follow can be used to follow any changes or updates made to a web page and pulls out the useful data. Tung said there are about 30 different page types on the web and Diffbot will be opening APIs for those over time, including profile pages, event pages and product pages. As those become available, expect more developers to give Diffbot a try.
Developers can grab more data from a variety of web sources and put them together into some interesting apps that should be useful for consumers. Tung said the company is profitable and is in the process of expanding beyond its five people. In addition, Diffbot provides a programmatic crawler that can be combined with page analysis APIs to extract and index databases of information from entire websites in real-time.
Diffbot's technology applies computer vision and natural language processing algorithms to web pages, executing all of the styling, scripting, and layout needed to produce visual information. The processes are CPU-intensive and users tend to submit content in bursts from news streams, social media channels, and other sources.
As a result, Diffbot has to be able to scale to handle frequent, real-time spikes in demand. The company runs its own data center and was using custom software to handle deployment and scaling. As we started to ramp up API call volumes, it was clear that we needed a better strategy for scaling our computing resources.
Diffbot handles hundreds of millions of API calls per month, but as a startup, it was not capital efficient to build out a large-scale on-premises infrastructure. Diffbot considered a variety of solutions, but chose Amazon Web Services AWS because of the scalability of the platform and the ability to leverage Amazon EC2 Spot Instances as a cost-effective way to purchase compute capacity.
Diffbot designed a solution that integrated the use of Amazon Elastic Compute Cloud Amazon EC2 instances with existing on-premises resources. Diffbot uses the compute-optimized c1. The high core count of these instance types means that multi-threaded code can utilize static objects more efficiently in memory. The higher clock speeds means that latency can be reduced.
The on-demand nature of some of its APIs means that traffic can spike throughout the day as new web pages are created across the web. Diffbot monitors resources with Amazon CloudWatch and utilizes Auto Scaling with custom predictive logic in order to scale up its analysis fleet during periods of high demand. This allows Diffbot to maintain high performance regardless of the amount of traffic it receives.
Diffbot processes hundreds of millions of web pages per month, and using Amazon EC2 Spot Instances lets the company flexibly prioritize and shift computing resources, depending on the level of requests. By running on the AWS Cloud, Diffbot is able to focus resources on developing cutting-edge machine learning algorithms, rather than worrying about hardware failure. Tung estimates that Diffbot can scale its infrastructure as needed in five minutes. The resulting level of reliability, performance, and scale gained as a result would have been impossible to achieve by building out our own servers.
What is your review of Diffbot? Though I have to specify the type of the website to increase the accuracy, they are the only tool that can extract you all the post from any blog Not limited to low-level Wordpress ones and display them in a structured way with author and date. Price is a little bit high and I wish they could classify websites automatically because if you try to scrape a news website while classifying as a discussion-type website, the output is going to be off.
Overall great product", Well, something similar is going on with the Semantic Web. The idea has never gotten very far, mainly because the burden of tagging all that content would fall to humans, which makes it expensive and tedious. But now it looks like the original goal of making digital content more comprehensible to computers might be achievable at far lower cost, thanks to better software.
Diffbot is building that software. This unusual startup—the first ever to emerge from the Stanford-based accelerator StartX, back in —is using computer vision technology similar to that used for robotics applications such as self-driving cars to classify the parts of Web pages so that they can be reassembled in other forms. In fact, companies pay Diffbot to analyze more than million unique URLs per month.
Building outward from its early focus on news articles, the startup is creating new algorithms that could make sense of many kinds of sites, such as e-commerce catalogs. The individual elements of those sites could then be served up in almost any context.
Imagine a Siri for shopping, to take just one example. What follows is a highly compressed version of my conversation with Tung and Davi. Xconomy: Where did you guys meet, and how did you end up working on Diffbot?
Mike Tung: I worked at Microsoft on Windows Vista right out of high school, then went to college at Cal and studied electrical engineering for two years, then went to Stanford to start a PhD in computer science, specializing in AI. When I first moved to Silicon Valley, I also worked at a bunch of startups.
I worked on search at Yahoo and eBay, and also did a bunch of contract work. I took the patent bar and worked as a patent lawyer for a couple of years, writing 3G and 4G patents for Panasonic and Matsushita. I first met John when we were working at a startup called ClickTV, which was a video-player-search-engine thing. It was pretty advanced for its time. Diffbot began when I was in grad school at Stanford [in ]. There was this one quarter where I was taking a lot of classes, so I made this tool for myself to keep track of all of them.
I would put in the URL for the class website, and whenever a professor would upload new slides or content, Diffbot would find that and download it to my phone. I always felt like I knew what was going on in my classes without having to attend every single one.
It was useful, and my friends started asking me whether they could use it. So I turned it into a Web service and … Next Page ». Wade Roush is the producer and host of the podcast Soonish and a contributing editor at Xconomy. Follow soonishpodcast. It turns out that on the modern Web, every page refresh changes the ads and the counters. You have to be a little more intelligent. A human being can look at Web page and very easily tell what type of page it is without even looking at the text, and that is what we are teaching Diffbot to do.
The goal is to build a machine-readable version of the entire Web. MT: It seems that every three years or so a new Semantic Web technology gets hyped up again. Because you are placing so much onus on the content creators, you are never going to have all of the content in any given system. So it will be fragmented into different Semantic Web file formats, and because of that you will never have an app that allows you to search and evaluate all that information.
But what if you analyze the page itself? That is where we have an opportunity, by applying computer vision to eliminate the problem of manual tagging. And we have reached a certain point in the technology continuum where it is actually possible—where the CPUs are fast enough and the machine learning technology is good enough that we have a good shot of doing it with high accuracy.
X: Why are you so convinced that a human-tagged Semantic Web would never work? MT: The number one point is that people are lazy. The second is that people lie. Google used to read the meta tags and keywords at the top of a Web page, and so people would start stuffing those areas with everything. The same thing holds for Semantic Web formats. Whenever you have things indexed separately, you start to see spam.
By using a robot to look at the page, you are keeping it above that. X: Talk about the computer vision aspect of Diffbot. How literal is the comparison to the cameras and radar on robot cars? MT: We use the very same techniques used in computer vision, for example object detection and edge detection.
If you are a customer, you give us a URL to analyze. We render the page using a virtual Webkit browser in the cloud. It will render the page, run the Javascript, and lay everything out with the CSS rules and everything. Then we have these hooks into Webkit that … Next Page ». For every rectangle, we pull out things like the x and y coordinates, the heights and widths, the positioning relative to everything else, the font sizes, the colors, and other visual cues.
In much the same way, when I was working on the self-driving car, we would look at a patch and do edge detection to determine the shape of a thing or find the horizon. MT: We have an ontology. Other people have done good work defining what those ontologies should be—there are many of them at schema. X: Do you actually do all the training work yourselves, or do you crowdsource it out somehow? John Davi: We have done a combination of things.
We always have a cold-start problem firing up new type of pages—products versus articles, or a new algorithm for press releases, for example. We leverage both grunt work internally—just grinding out our own examples, which has the side benefit of keeping us informed about the real world—but yeah, also crowdsourcing, which gives us a much broader variety of input and opinion.
We have used everything, including off-the-shelf crowdsourcing tools like Mechanical Turk and Crowdflower, and we have build up our own group of quasi-contract crowdsourcers. Our basic effort is to cold-start it ourselves, then get an alpha-level product into the hands of our customer, which will then drastically increase the amount of training data we have.
Sometimes we look at the stream of content and eyeball it and manually tweak and correct. In a lot of cases our customer gets involved. If they have an interest in helping to train the algorithm—it not only makes it better for them, but if they are first out of the gate they can tailor the algorithm to their very particular needs.
X: How much can your algorithms tell about a Web page just from the way it looks? Are you also analyzing the actual text? Article pages, people pages, product pages, photos, videos, and so on. So one of the fields we return will be what is the type of this thing. Then, depending on the type, there are other fields.
For the article API [application programming interface], which is one we have out publicly, we can tell you the title, the author, the images, the videos, and the text that go with that article.
0コメント