Sheep Spidering SL Grid for New Search Service
The Electric Sheep Company (sponsors of this blog) launched a new beta search service for the virtual world of Second Life today, at search.sheeplabs.com, according to Electric Sheep Christian Westbrook. What’s unique (as far as I know) about this service is that it doesn’t rely on users to manually list their products but instead spiders the SL Grid to automatically collect information about items marked “for sale.” (Read more about how it works on the service’s About page.) The service allows avatars to opt out of the system, or to list all items they own, and doesn’t crawl private islands. Results are returned with a teleport link, price, object creator and owner, and description. According to the Sheep, it also puts less load on the system than a single avatar, so it shouldn’t create much lag.
Having someone spider the SL Grid is something I’ve been looking for for a long time, so I’m looking forward to seeing how this works in practice. Having users manually list objects, as the many other SL search services do, is a far from comprehensive solution, but it’s been the best we’ve had until now. If all goes as planned, this should push SL search forward by leaps and bounds.
The one thing the service’s About page doesn’t say is how tags or search terms are determined for objects. With any luck, and if the Sheep are smart, they’ll find a way to incent users to list tags in the object name or description field within SL, which seem like the only metadata-capable fields they’d have access to. I’m not entirely sure this would work, but I think a Grid crawler would be able to access one of those fields. Incentive comes from the fact that adding tags would make your objects easier to find — just as adding metadata tags to a Web page makes it easier to find in a Web search engine. Remains to be seen if this is possible, or if it’s something the Sheep are interested in, but until there’s some solution to the metadata issue, even a comprehensive spidering of the Grid will produce less than optimal results. [UPDATE: It also doesn’t say how search results are ordered, I now realize.] Still, this should be a great advance in Second Life search, where advances are sorely needed. Great stuff.
Feeds
Interesting they seem to have changed the account running the index bot. Over the weekend, I could have sworn that indexing was being done by “Search Shepherd”, but now it appears to be “Grid Shepherd.”
I would imagine it would take many bots to scale this effectively, get ready for the ban equivalent of robots.txt.
[…] Just as Mark Wallace points out in his blog post, I too am very curious to see how this search service changes the behavior of naming and describing objects on the Second Life grid. It will only benefit content creators to use the meta data fields available to them to maximize indexing potential. […]
ok, this is scary.
From the about page:
‘Because SheepLabs places utmost importance on privacy considerations, we have limited the service to products. However, if you would like to remove objects you own from our service, please visit the SheepLabs headquarters and click on the search.sheeplabs.com sign to change your privacy preferences:’
This is an opt out scheme, not an opt-in scheme? This is not good, this should be optional buy-in, not a ‘opt-out if you want’. I have never signed up to you, or allowed you to use my IP?
Also, does this contrevene the T&C’s of LL, are they endorsing this?
Is this something LL should have done in the first place?
Who are you selling the search data too, and why are we being snooped? If this is data, it should be shared and used by LL - not a developer!
This is like the copybot fiasco! A total disregard for privacy too!
Aha! OK the way around this:
Just like any annoying stalker or griefer, all you have to do is ban the following from your parcel:
Grid Shepherd
Seems like the best banning I’ll ever do, stop trying to herd a girl - jeez!
Sorry, er, whats the problem with this? It returns stuff for sale, so people looking to buy it can find it, and pay you for it. Surely if you didnt want people randomly coming to your land to give you money, you wouldnt have said.. hey, you can all come here, if you want, and buy this.. all this search thing is doing is taking your offer and making it easier for people to hear. We all know the grid search is screwed, and this is way way faster than that too. So, I’m all for it. There’s no downside here. They arent people operating this, they arent giving particular preference to corporate sponsors or clients, where’s the downside?
when I said “they arent people operating this” I meant, its a bot, not a person going around going, oo she was nice, she’s going at the *top* of the list. its a robot. it doesnt give a monkeys. as it should be.
Just to clarify :)
Interesting that electric sheep are getting into this. Last week I picked up a really neat hud called Slicr that tags and searches places. Cool design too, worth checking out.
I’ve blasted this greedy ESC spider here:
http://www.secondlifeherald.com/slh/2007/04/the_greed_sheph.html
and here:
http://secondthoughts.typepad.com/second_thoughts/2007/04/the_electric_sh.html
The Internet has spiders that scrape data from websites which are human artifacts that humans put up with a set of conventions that indicate that they want them searched.
In Second Life, people didn’t want them to be searched. They live and work inside Second Life like a world; they don’t live and work inside a Google search or a webpage. The social and political dimensions dictate different behaviour by greedy marketing scrapers like ESC.
Portraying this as “opt-in” is HUGELY misleading. It isn’t. You can’t take your land out of it. You can only take your owned object out of it — after the fact.
Saying it’s “only private islands” is a lie — the private islands are in fact in it — I see them in there for sure. Type “Ravenglass Realm” the name of one of my private islands and see all my tenants stuff, not intended for sale, now out in public. Wrong, wrong, wrong.
And even if that’s a glitch…Why should private property on the mainland have to be scraped by greedy developers eager to capitalize on market data and be the first to offer it to their clients? Why should private islands get to be free of that intrusion, just because the people are wealthier?
Are mainlanders guinea pigs for marketers and big business? Why do they get to do this, and no other company. Serious questions needs to be asked about LL’s relationship to this.
I didn’t opt in MY LAND to be scraped — and that’s what happened. Hunderds of my tenants now have their stuff visible and put into a data base they didn’t ASK to be in. They have to find out AFTER the fact that they must scurry and either opt out of the dbase or turn off anything for sale — which they’re now finding out about.
I think SEARCH is a public utility that shouldn’t be accruing to the financial benefit, marketing research, and reputation enhancement of any
one company.
Er, Apple, people did not KNOWINGLY put everything to sale FOR THE PUBLIC. You’re dead wrong on this. There are many cases of items in SL that automatically stay on sale when you take them out of inventory. People put TVs to sale to undeed them. There are many examples of things for sale that people don’t want for sale TO THE ENTiRE PUBLIC. Berating them and clubbing them over the head with this is just frigging retarded; they didn’t ASK to be in this.
It’s only an opt-out if you get to CHOSE whether to show up.
What would have bee
*what would have been appropriate is to:
1) announce that Grid Shepherd would be visiting stores and any other parcels people would like to have things show up in search for
2) offer people the opportunity to ban Grid Shepherd off their land
3) warn them that anything for sale, accidently or on purpose, will show up, and be vulnerable to theft
Naturally, that wouldnl’t have given them what they greedily sought: a kick ass sample of 422,000 odd objects to impress their clients with who are lusting after all our marketing data.
>bluesapphire on April 9th, 2007, at 4:54 pm:
Aha! OK the way around this:
Just like any annoying stalker or griefer, all you have to do is ban the following from your parcel:
Grid Shepherd
Seems like the best banning I’ll ever do, stop trying to herd a girl - jeez!
BlueSaphhire, please try to be aware of how this won’t work.
You can scan things on a parcel even being banned on it; if you can buy anytying, that means you can camera zoom to see it, that means this avatar bot can get to it — the radius might be 96 m2 as it was with slstats.com
Everything is ALREADY in there. So you can prevent future scanning, but you can’t remove what you have in there already unless you go and try to get their opt-out page now to work, which is opting out by name of creator or object but NOT by your land.
It remains free and capable of scanning and scraping anything for sale on your land. This will likely escalate to scanning and scraping ANYTHING AT ALL and AVATARS TOO on land.
You are not telling the full story about it, Apple, you’re wrong.
http://www.secondlifeherald.com/slh/2007/04/the_greed_sheph.html
I am one crazy bitch.
http://www.ftrain.com/robot_exclusion_protocol.html =D =D =D
Also: is Grid Shepard available for escort services?
“Having someone spider the SL Grid is something I’ve been looking for for a long time,”
Really, Mark? Where were you the last time land scanners were shouted down?
LOL Hiro!
OR better still!
My sponsor has been interested in Land Scanners for a long time, I’m therefore compelled to write about it here.
p.s.
LET HIRO AND BOLIVER RUN THE SLCC!
SL内を実際に情報収集する検索システムがベータテスト中…
3pointD.comにオモシロイ記事が出ています。「新しい検索サービスのために羊がSL世界にクモを放っています」。他の検索システムとの大きな違いは実際にSL内をクローラーが自動的に調…
Yeah, you guys really should’ve given Prok a few more days to stuff that freebie table Barnesworth Anubis made back into inventory, lol.Of course, now she wont get the L$ she was selling it for, but that’s progress!
[…] As you might imagine there are a variety of responses to this. Over on 3pointD Mark is happy to see it arrive, some of those that comment are less sanguine. Over on SLH Prokofy is sounding uncharacteristically against making money, but characteristically paranoid that it’s all a ploy to exploit him. Potentially more worryingly for the future of the project, Anshe has banned it from all of her land for “violating the covenant.” […]
Hello Mark - I hope you are well. An interesting post which pressed one of my buttons.
Not everyone’s junk drawer is worth cataloging (mine isn’t). I will be curious to see how the ESC manage the qualitative aspect of this service. In addition to potential coverage/access issues from banning & opt-outs, it will also be interesting to see how the ESC deals with the Grid Sheppard in terms of the many forms for griefing available against it, some potentially further compromising the qualitative aspect.
I enjoy reading your blog.
Peace,
HatHead.
HatHead - Interesting thoughts. The griefing aspect against the bot is definitely something I think needs to be considered, lest it render the service as useless as the current search system.
As far as listings of ‘everyone’s junk drawer’, it should be noted that only items that are marked for sale are listed on the site at this point. THere is an option for people to opt in *completely* and have all of their items listed if they really want to… but as you mention… would you really want to?
Ultimately, this service is just a tool. I personally can’t imagine anyone who would purposely go out of their way, as some suggest, to buy marked items off someone’s private property, and I certainly don’t believe ESC ever intended such to happen.
I *do* believe business owners and their customers can use it as a tool for finding what they need, but I also believe it is not the demon that some make it out to be and could actually be used to *help* people make sure they’re *not* inadvertently leaving their objects up for sale.
[…] 3PointD […]
@Hiro: Not sure what you’re asking there. I like the idea of the Grid being spidered, with an opt-out. I’m not sure if there’s a more effective way to build a truly useful search engine.
Hang on - Grid Shepherd is a No Payment Info
It picked up something on a parcel that bans No Payment Info, listing it
as something belonging to an adjacent parcel.
Which makes me wonder how effective it is to ban it from a parcel …
Buckly> Probably not very. Consider this - any time you rez up anything within your draw distance, you’re gathering information just as Grid Shepherd would. About what the object is, what permissions are on it, who owns it, whether it’s for sale.
You don’t have to be standing on a banned parcel to get information about what’s in it. Your client just has to load it up.
Aki - you and I know that.. but..
What it means is that if people think they can just ban ‘Grid Shepherd’ from their parcel, and they wont get scanned.. they need to think again. They would still be scanned from their neighbor’s parcels.
Add to that the fact that people have to proactively TP over to ESC to get removed. The tech is good. The deployment is backwards.
Bucky, it’s not effective. Banning it from a parcel only makes sense if you hae 16 sims or bunches of parcels to keep it from grabbing everything in a 96 m2 radius. I believe it can sweep in that radius, which is why it doesn’t matter if you ban it.
and opting out merely removes the objects created or owned by you from public view, it doesn’t prevent the Sheep from holding your private and proprietary data in their data bases.
Furthermore, you can’t take your land out — it keeps scanning and running it, regardless of whether you want it in or not.
Aki, again, your notion that something is “for sale” because it has the price tag left on it is very mistaken. Stuff is put on sale by accident; left on sale; deeded to the group for $0 or bought back for $0; given to a friend for $1; taken out of a box still showing the sale. So that is not “showing it for sale”.
Your argument is akin to saying “the car has the carkeys in it so I can joyride it and that’s not a crime” or “the date is drunk and passed out so I can rape her”.
Sorry, no sale.
>Consider this - any time you rez up anything within your draw distance, you’re gathering information just as Grid Shepherd would. About what the object is, what permissions are on it, who owns it, whether it’s for sale.
>You don’t have to be standing on a banned parcel to get information about what’s in it. Your client just has to load it up.
Um, I don’t then load all that up to a searchable public data base though, do, I, and I don’t traverse the grid accumulating everybody data and scraping it and uploading it to use for my company’s own benefit exclusively.
Seriously, trying to rationalize away people’s legitimate concerns about privacy and exploitation just doesn’t fly. People aren’t going to buy this word-salading you’re doing about this.
I think either you like this, or you dont. Sheep is going omg, we’ve built a really good search engine!
Profky is going, omg they’ve built a really good search engine >;(
I’m with the sheep. This helps me find things i want to buy. And this anti privacy stuff, its not like they’re publishing my social security id. its a few prims, that I’ve set for sale, big frickin deal.
Oh, Grid Shephard is totally fucked. When I tried it it listed three bdsm sex items as for sale on my land which do not belong to me and are actually on an adjacent plot. Prima facea under English law should I not be anonymous there’d be a strong case for defamation there. Furthermore as is suggested here if I ban it and it still comes up with incorrect results like the above because it cannot tell precisely where objects are - and the results indicate that this is the case - then My Learned Friends could have a field day.
I’m beginning to wonder what it is with the kudos paid to ESC. It’s not even as if they’ve done anything clever - the necessary core code is publically available in the FeatherTail project
Hate to be the bearer of bad news Prokofy, but what you consider ‘private’ data in Second Life - Any and everything that actually gets rezzed - is not private. It is public.
There is no debate on this point - none at all. Private information is that which cannot easily be viewed without illegal means: You account/payment information on LL’s web page for example. That requires your user name and password - it is private.
In addition there is no one ‘living’ in second life. There is no hope of an argument on this as I have yet to see someone linking to a news article stating that someone uploaded their entire consciousness to the Internet.
Please, separate second life from real life long enough to return to reality.
So based on everything I’ve read here (mostly from Prokofy Neva) is that this is really only bad if you’re behaving dishonorably by selling things that you got for free (like Prokofy Neva). I’ve already used it to find some stuff - hair, clothing, etc., that I otherwise likely never would have without this tool. The only tool I could do without, thus, far, is Prokofy Neva.
Maybe the spider could do something similar to Google. If you don’t want it to search your site, you rez an object with some special code given by the ESC. Just like google has you put a text document with some special code in case you don’t want their spider to catalog that directory.
This could also be used a few more ways. When the spider comes by, the object could already have all the scanned data so it can operate quicker.
Fellow Developers of Web3D technologies and users..
Ive followed this thread with interest over the week.
Ive just used the ESC search again today, Sunday - I now believe ESC is overstepping privacy and basic LL TOS lines and should shut down the search beta until it is OPT IN and properly functioning.
Last week when I tested a search string, I saw items of mine that were only “on display on the open mainland and marked BY me for sale” and thus listed. This was “almost fine”, but i as a business i would appreciatte the usage of my marks of trade be vetted by us. The items were then listed IN the correct parcels of our partner vendors.
Today, i searched again, NOT ONLY were many of the parcels listed as “locations of sales” incorrect- thus implying business relationships that do not exist, BUT I found Items on ours PRIVATELY(under LL- TOS) sold to Individuals LISTED on their PRIVATE, BAN ENABLED
(SL-TOS) locations. These items were sold as NON SALE to fellow SL users under TOS of LL. I dont know why these items would show up publically on the ESC owned website, they shouldnt, and if a customer desires “private” visual ownership/display of a private purchase from C3(other than LL records - “private” under TOS) they SHOULD recieve such terms as generally understood upon by the TOS and more importantly by the promoted terms advertised by LL.
As a brand seller in SL i welcome easier ways to sell/promote my products- with BTW are under IP license from Cube productions Inc. seperate and additional to any LL TOS.. BUT i do not welcome a public listing of customers or false listings of possible business relations….I have no specific agreement that allows this to be pubically displayed by Linden Labs or ESC.
Imagine if Google PUBLICALLY published all owners and keys to all bathrooms of owners who subsribe to PLAYBOY magazine.. Such a tame example, but think OF the ramifications……Does google publish the address of the store and name of owner of every location that has a specific product just “lying around” in the back room. I would say not. A “stray” copy of “Playgirl” laying in a back room at the “Whitehouse” (owned, planted or misplaced by anyone)could in our culture today provide never ending irrelevancy to the public arena. Now imagine all the names are false. The magazine was actually “The Nation” not Playboy….and it was not in the White House but next door…in a guards shack.
This is the current “public functionality” of the ESC search engine as i have used it today.
The Search sytem being “played with publically” by the ESC is not a positive step for metaverse development as deployed. I would ask as one developer to another, that the search BETA be pulled UNTIL a fully working and OPT IN service is offered by ESC and or LL.
I would hope that as a professional company that they would treat others the same way.The methods they are using for the “fast” dominance of search, for the obvious commercial benefits for their investors and owners, seem to this developer as going beyond the expectations of users of the Linden TOS. Thus Linden lab itself should also be involved with the removal of this “service” until it is properly vetted as deployable and in the spirit of the current TOS that many paying customers/clients are invested in.
In today’s culture and tommorrows “metaverse”, We “are” our items owned or licensed — and we should expect both from a private company like Linden Lab and their Partners like ESC a matter of respect “for their consumer” and their fellow “business colleagues.” A better understanding of the “new media” they are taking credit for developing should be the first step for both commercial entities.
Ive stated LL is NOT a community or a governemnt or any of the “visions” accorded to it by the bloggers who frequent these circles. It is though, as is ESC, a commercial entity under laws and prevailing “community” responsibilities. In this specific case, I believe questionable actions have occured and should be corrected.
If “this new internet way” so often spouted here is to replace the US Constitution or the current US state and federal laws that suggest how corporate citizens behave with others and the public, then so be it. May I suggest the owners and investors of these companies be the first to open their bathroom doors to the public.
Respectfully,
Larry Rosenthal
Cube3.com