PDA

View Full Version : New Search Engine


arn
Nov 18, 2011, 02:07 PM
We've installed a new search engine which hopefully will address some of the limitations with our previous solution. I forget all the issues that we had, give the new search a try.

Detailed instructions are here: http://forums.macrumors.com/faq.php?faq=searchlight_search

Feel free to post any feedback / here.

arn

macbookairman
Nov 18, 2011, 02:09 PM
I just noticed today that if I highlight any part of a post, it brings up a magnifying glass. When I click the magnifying glass, it searches the forums for threads/posts related to the highlighted phrase. This seems pretty useful.

I'm wondering if this is new, or if it was just added. I don't remember noticing it in the past, and a quick search didn't show anything related to it.

Screenshots:

EDIT: Just noticed arn posted a thread about this just minutes ago. Seems its related to a new forum search engine.
Thread: http://forums.macrumors.com/showthread.php?t=1278345
More info: http://forums.macrumors.com/faq.php?faq=searchlight_search

greganpace
Nov 18, 2011, 02:14 PM
I think it is new. I noticed it today for the first time as well. I like it. It should prevent a lot of the duplicate posts because it is easier to search for topics. It has made it very hard to actually highlight something if I'm trying to quote it though. It makes the page lag.

r2shyyou
Nov 18, 2011, 02:14 PM
Seems to work well though I've obviously only been playing with it for a few minutes.

Is the "Similar Threads" thing at the bottom of the page in the attachment involved in any way? I just noticed this today.

arn
Nov 18, 2011, 02:18 PM
ya, we've enabled similar search. it might take a little longer for the db to fill out.

arn

saving107
Nov 18, 2011, 02:22 PM
Cool

arn
Nov 18, 2011, 02:30 PM
I think it is new. I noticed it today for the first time as well. I like it. It should prevent a lot of the duplicate posts because it is easier to search for topics. It has made it very hard to actually highlight something if I'm trying to quote it though. It makes the page lag.

Ya, I wasn't entirely sure about it. You can turn it off in User CP -> Edit Options -> Contextual Search

arn

TECK
Nov 18, 2011, 02:39 PM
I think it is new. I noticed it today for the first time as well. I like it. It should prevent a lot of the duplicate posts because it is easier to search for topics. It has made it very hard to actually highlight something if I'm trying to quote it though. It makes the page lag.

More how the Contextual Search (http://forums.macrumors.com/faq.php?faq=searchlight_search) works. Scroll down near to the bottom of the page.

r2shyyou
Nov 18, 2011, 02:46 PM
Not sure you want this kind of "feedback" but I read the whole Axivo Searchlight FAQ page that you provided the link to and found a spelling error (in red below):

Why my recent posts don't show up rightaway in search results?Each new post is scheduled to be stored into a search index, at regular periods of time. Therefor, if you created a new post, it will take approximately 10 minutes before your post content will be recorded and displayed into search results.
For example, while viewing the Advanced Search (http://forums.macrumors.com/search.php) or Search Results pages, you will notice at the top of the page a message like:
"The search index was updated 8 minutes ago." That means the next index update will be executed in approximately 2 minutes.

Note: The forum administration might change at what interval of time the index updates can occur.

In other news and perhaps more on-topic, the Contextual Search is cool and will hopefully prove useful for me (and others, of course) when I'm looking for related threads/posts, however, it seems to be a bit buggy where it shows many duplicates of the same words in the search window (see attachment below). Here's an excerpt from that attachment:

...available updates. ClickClick "App StoreStore." ClickClick "UpdatesUpdates" if it is not already open. SelectSelect "Update All … entering anything. Screen displaysdisplays "All Apps Are Up to Date...

KingYaba
Nov 18, 2011, 02:48 PM
I like it.

Firestar
Nov 18, 2011, 03:33 PM
I highlight text to mark where I'm reading, so this is getting turned off.

rick snagwell
Nov 18, 2011, 03:49 PM
i love it!

arn
Nov 18, 2011, 08:14 PM
In other news and perhaps more on-topic, the Contextual Search is cool and will hopefully prove useful for me (and others, of course) when I'm looking for related threads/posts, however, it seems to be a bit buggy where it shows many duplicates of the same words in the search window (see attachment below). Here's an excerpt from that attachment:

this should have been fixed.

arn

PlaceofDis
Nov 18, 2011, 08:29 PM
i was so confused at first with the contextual search. thank you for allowing us to disable it.

Doctor Q
Nov 19, 2011, 02:03 AM
Note that you can double-click a word to highlight it and reveal the contextual search icon, or drag to select a phrase and reveal the contextual search icon.

miles01110
Nov 19, 2011, 03:45 AM
Did the advanced search get a makeover as well? When I search for all threads with zero replies since my last visit there are still a few at the tail end that have multiple replies. This is an improvement over the previous system, which would give me over 800 threads total, most of the tail end of which all had replies.

simsaladimbamba
Nov 19, 2011, 08:00 AM
Does the Advanced Search for threads with zero replies still work, as I get a "Sorry - no matches" result now, or is that due to using EDGE connection speeds on an Android telephone?

maril1111
Nov 19, 2011, 08:11 AM
Great update, working flawless so far!

TECK
Nov 19, 2011, 10:13 AM
Does the Advanced Search for threads with zero replies still work, as I get a "Sorry - no matches" result now, or is that due to using EDGE connection speeds on an Android telephone?

Try this search (http://forums.macrumors.com/search.php?query=search&exactname=1&starteronly=0&childforums=1&titleonly=0&showposts=0&searchdate=0&beforeafter=after&sortby=lastpost&sortorder=descending&replyless=1&replylimit=0&searchthreadid=0&saveprefs=1&quicksearch=0&searchtype=0&nocache=0&ajax=0&userid=0&operand=0&ctextprocess=0), it will return threads with zero replies.

Doctor Q
Nov 19, 2011, 10:32 AM
Thanks, TECK. That works for me.

Here are the relevant parts to this type of search, to find the reply-less threads in whichever forum(s) you are interested in.

312790

GGJstudios
Nov 21, 2011, 12:14 PM
I think something's broken in the new forum search. I frequently search for threads with keywords in the titles, rather than in posts. This doesn't work now. Here's a search I just performed, searching within thread titles only:
313069
And here's the search I did immediately afterwards, only changing the "Search Titles Only" to "Search Entire Posts". As you can see, there are thread titles that met the criteria of the first search, but they were not identified by that search.
313070

BC2009
Nov 21, 2011, 04:14 PM
If you try to search for posts containing a user name to find replies to a user, its not hitting them all. Does it not yet everything indexed?

TECK
Nov 21, 2011, 05:30 PM
I think something's broken in the new forum search. I frequently search for threads with keywords in the titles, rather than in posts. This doesn't work now. Here's a search I just performed, searching within thread titles only:
313069
And here's the search I did immediately afterwards, only changing the "Search Titles Only" to "Search Entire Posts". As you can see, there are thread titles that met the criteria of the first search, but they were not identified by that search.
313070
Works for me, I get results galore for battery in title.

If you try to search for posts containing a user name to find replies to a user, its not hitting them all. Does it not yet everything indexed?
What do you mean search for posts containing a user name to find replies?
If you search for keyword TECK, it will look for that into threads title and posts content. It will not check if the user TECK made any new posts.
The easiest way to see threads started by a user or his latest 500 posts is to click on user profile and select the tab Statistics or simply click on post username and select the "Find more posts" option. For example, here are all my posts (http://forums.macrumors.com/search.php?do=finduser&u=642422).

GGJstudios
Nov 21, 2011, 06:31 PM
Works for me, I get results galore for battery in title.
Please post a screen shot of your search results. Before you do, make sure you're using the same settings I used, which are circled in my post.

arn
Nov 21, 2011, 09:04 PM
I can replicate the results.

If you set the time for

"Yesterday and Newer" and "Title Only", you get no results.

arn

GGJstudios
Nov 21, 2011, 09:07 PM
I can replicate the results.
If you set the time for
"Yesterday and Newer" and "Title Only", you get no results.
That's exactly the search I use daily. I haven't tried all the different timeframes, but I did notice if I use "Any Date" it works.

Frosticus
Nov 22, 2011, 07:28 AM
If you try to search for posts containing a user name to find replies to a user, its not hitting them all. Does it not yet everything indexed?

I found this to be a new problem too.

I used to be able to easily find posts where I (or another specified user) had been quoted simply by searching. I can no longer do this with the new search engine :(

TECK
Nov 22, 2011, 08:40 AM
I can replicate the results.

If you set the time for

"Yesterday and Newer" and "Title Only", you get no results.

arn

This is a confirmed bug, I'll have it fixed in 10min.

GGJstudios
Nov 22, 2011, 08:56 AM
This is a confirmed bug, I'll have it fixed in 10min.
Your 10 minutes are up. :D By the way, who are you?

TECK
Nov 22, 2011, 10:15 AM
Your 10 minutes are up. :D By the way, who are you?

Hi,

I emailed the fix to arn, is up to him now when he will upload the files. :)
I'm the developer who created the search product (http://www.axivo.com/searchlight/) for vBulletin.

arn
Nov 22, 2011, 10:58 AM
fix has been uploaded.

arn

TECK
Nov 22, 2011, 11:04 AM
fix has been uploaded.

arn

Let me know if the new search criteria (http://forums.macrumors.com/search.php?query=battery&exactname=1&starteronly=0&childforums=1&titleonly=1&showposts=0&searchdate=1&beforeafter=after&sortby=lastpost&sortorder=descending&replyless=0&replylimit=0&searchthreadid=0&saveprefs=1&quicksearch=0&searchtype=0&nocache=0&ajax=0&userid=0&operand=0&ctextprocess=0) works properly now.
Edit: Please note that the search criteria bases its date line on the initial thread date (otherwise first post made into thread), if you search in titles only. So, if you search for "battery", it will look into titles for threads started since Yesterday or more recent.

TECK
Nov 23, 2011, 10:10 AM
I found this to be a new problem too.

I used to be able to easily find posts where I (or another specified user) had been quoted simply by searching. I can no longer do this with the new search engine :(

Quotes are ignored, in order to save disk space into indexes. I understand your logic behind this but is just redundant information that will not change. If you want to see if someone quoted you, simply subscribe to a thread and follow the replies or ask arn to install a quote notification system that alerts you on that event.

Even if you will be able to search into quotes for your name, you will not find all results anyways since the search works like Google Search. It puts weight on most found keywords, than it picks most relevant 1000 threads and orders them by date.

arn
Nov 23, 2011, 10:15 AM
or ask arn to install a quote notification system that alerts you on that event.


... looking into this. :)

arn

GGJstudios
Nov 23, 2011, 10:16 AM
Let me know if the new search criteria (http://forums.macrumors.com/search.php?query=battery&exactname=1&starteronly=0&childforums=1&titleonly=1&showposts=0&searchdate=1&beforeafter=after&sortby=lastpost&sortorder=descending&replyless=0&replylimit=0&searchthreadid=0&saveprefs=1&quicksearch=0&searchtype=0&nocache=0&ajax=0&userid=0&operand=0&ctextprocess=0) works properly now.
Edit: Please note that the search criteria bases its date line on the initial thread date (otherwise first post made into thread), if you search in titles only. So, if you search for "battery", it will look into titles for threads started since Yesterday or more recent.
Yes, it appears to be working now. Yes, I'm aware that when searching by thread, the time criteria returns results based on the date of the thread (1st post). If searching by post, the results are based on the date of the post. I use the thread search to find new threads that were started in the past day/week that dealt with a particular topic.

BC2009
Nov 23, 2011, 10:59 AM
I found this to be a new problem too.

I used to be able to easily find posts where I (or another specified user) had been quoted simply by searching. I can no longer do this with the new search engine :(

I could not even find that you replied this. This new search engine sucks. Still broken.

To replicate please go to the search drop-down menu or advanced search screen. Enter a user name (like my own: "BC2009") and search for "Posts". This should return all posts that contain my user name (typically replies to my comments or people insulting me or in rare rare cases complimenting me). Anyway, it shows only 39 results. I know that the results were previously in the hundreds.

What gives?

UPDATE: The problem appears to be when the user name only appears as the first part of the post within the bracketed "QUOTE" tag. At least that is what I am seeing. Posts like this that include a user name outside the quotes are getting found as well as those that begin with the "Wirelessly posted...." stuff. Sure would be nice if somebody from MR replied to this. Thanks.

BC2009
Nov 23, 2011, 05:26 PM
Quotes are ignored, in order to save disk space into indexes. I understand your logic behind this but is just redundant information that will not change. If you want to see if someone quoted you, simply subscribe to a thread and follow the replies or ask arn to install a quote notification system that alerts you on that event.

Even if you will be able to search into quotes for your name, you will not find all results anyways since the search works like Google Search. It puts weight on most found keywords, than it picks most relevant 1000 threads and orders them by date.

Lame. The worst thing any developer on my team can tell me is "we are not giving the user what makes sense because for some technical reason it does not fit with how we implemented things". Saving disk space does not seem to be a valid reason for removing such a valid use case. I must admit that i prefer the system that The Verge uses that lets me see all my posts and which have replies, but that system connects each reply with the original post with more than just contained text in a block quote. Not sure if the MR forums do this.


... looking into this. :)

arn

It would be best if it hung off your "profile" like on The Verge. A list of all your posts and with each post an indication of the number of replies (and preferably a link to each reply).

Frosticus
Nov 24, 2011, 08:35 AM
Quotes are ignored, in order to save disk space into indexes. I understand your logic behind this but is just redundant information that will not change. If you want to see if someone quoted you, simply subscribe to a thread and follow the replies or ask arn to install a quote notification system that alerts you on that event.

Even if you will be able to search into quotes for your name, you will not find all results anyways since the search works like Google Search. It puts weight on most found keywords, than it picks most relevant 1000 threads and orders them by date.

But I don't want to have to subscribe to every single thread I post in to be able to see if I have been quoted (i.e. if another user has responded directly to something I have said). I will end up subscribed to far too many threads, I never had to do this before and therefore this to me is a step backward in functionality.
Please fix!

TECK
Nov 24, 2011, 06:37 PM
Lame. The worst thing any developer on my team can tell me is "we are not giving the user what makes sense because for some technical reason it does not fit with how we implemented things". Saving disk space does not seem to be a valid reason for removing such a valid use case. I must admit that i prefer the system that The Verge uses that lets me see all my posts and which have replies, but that system connects each reply with the original post with more than just contained text in a block quote. Not sure if the MR forums do this.

What the developers are trying to tell you is that you cannot have everything you want in life. On top of that, you are not even open to listen or understand what is the proposed solution. FAIL

But I don't want to have to subscribe to every single thread I post in to be able to see if I have been quoted (i.e. if another user has responded directly to something I have said). I will end up subscribed to far too many threads, I never had to do this before and therefore this to me is a step backward in functionality.
Please fix!
The current search behavior will not be changed. The proper way is NOT to search for keywords on text that should not be searchable to begin with (redundant quote text, already present into database) but rather be alerted if someone quoted you in a thread so you can directly view that quote and not be forced to search for it.

GGJstudios
Nov 26, 2011, 12:02 PM
OK, here's another wrinkle. If I search for threads started by a person, it doesn't show all the threads. Whether I use the "Find all threads started by xxx" link in the profile, or whether I use Advanced Search to "Find Threads Started by User" with a date range of "Any Date", it doesn't produce accurate results. For example, when I search for my own threads that I've started, it only shows 2. While I don't start many new threads, I know I've started much more than that over time.

BC2009
Nov 28, 2011, 12:56 PM
What the developers are trying to tell you is that you cannot have everything you want in life. On top of that, you are not even open to listen or understand what is the proposed solution. FAIL


FAIL? Seriously, let's bring this back to a constructive discussion.... I think you missed my point. It worked before, and now it does not work and there is no alternative in place. So "you can't have everything you want in life" actually translates into "you can't have what you had before because we forgot to think of this use case" The #1 wrong answer to any valid customer use case has got to be:

- "you cannot have everything you want in life"

The #2 one is:

- "we cannot do it because our implementation did not consider that use case and now it is too hard"

Regarding, alternative solutions -- I am all for that -- you would be a hero if you could implement it quickly. But this new search feature went live before the new alternatives were in place. I am guessing the MR forum does not actually store links to original posts that were quoted. More likely it is implemented as a linked list of database records containing the comment text and a thread ID, and within the comment text is the quoted segments from the original poster including a link back to the original post (i.e.: it is unstructured data).

If that is case, then the only way to implement it is to recognize the links back to original posts via pattern matching, extract those links and maintain a reference table of "quoted posts" for each posts. From that the reverse-lookup can also be done. So the table would contain:

thread_id, poster_user_id, post_id, reply_to_user_id, reply_to_post_id

If you had this table you could lookup all posts by a given user, all posts in reply to a given a user, all posts in reply to a specific post. The construction and population of this table requires some unstructured text analysis on the existing database, but its not outside the realm of the same code being used to exclude the quoted blocks in the first place. Essentially, exclusion of quoted blocks is just pattern matching and ignoring what is being matched. Instead you'd be looking for pattern matching and then analysis of the block for links to other posts.

It would be incredibly cool if this site offered the new search functionality along side new functionality to click on your user name and bring up all your comments with reverse links back to everybody who quotes your comment with a link back (The Verge does something close to this). However, this was not the case. Instead the new functionality went live BEFORE the alternative was even coded (or possibly even considered).

You would be a hero if you could pull off this new functionality in a week. But until this feature is up users are crippled in their ability to monitor responses to their posts. Very few of us with real jobs can sit around all day and follow an entire thread as it develops. To summarize the requirements:

1) Need ability to find all posts made by a user (including myself)

2) Need ability to find all posts made in reply to a user (including myself)

3) Need ability to find all replies to a specific post.

4) Need all of the above without annoying messages being sent to my email to "alert me" -- this needs to be Web-based functionality that is produced on demand.

I know I am not the only user who wants this. Keep us posted on progress.

PS: I get that you worked hard on this new search engine and sucks to have people disregarding the improvements and focusing on the deficiencies. Perhaps I was a little overly critical, but I feel you have overly minimized the use case that is being brought up here. This is how it goes in software development when you release something that takes away support for valid use cases. Randy Ubillos with Apple is probably dealing with much of the same feelings over Final Cut Pro X right now -- for a guy like me it FCPX looks great because I was not a previous user who lost functionality. However, to those who leveraged something that is now gone it is extremely frustrating. My MacRumors participation experience has been seriously degraded since I cannot see how others replied to me or to others users whose posts I find interesting. Your "alerting solution" is not desirable because it likely would not let me be alerted when other users get responded to and also because I don't want an email alert -- I want a listing of all activity I am interested in. Previously the "search" feature served as a proxy for that. The proxy is now gone and MR just became far less usable for me and for others. Still easy to read the articles -- not so easy to participate in the discussions without wading through everything.

TECK
Nov 28, 2011, 02:54 PM
OK, here's another wrinkle. If I search for threads started by a person, it doesn't show all the threads. Whether I use the "Find all threads started by xxx" link in the profile, or whether I use Advanced Search to "Find Threads Started by User" with a date range of "Any Date", it doesn't produce accurate results. For example, when I search for my own threads that I've started, it only shows 2. While I don't start many new threads, I know I've started much more than that over time.
Thanks for the report, I will look into it and let you know.
Edit: The results look accurate to me, on my development server. Can you start a new thread, wait 10min and display new threads started by you (http://forums.macrumors.com/search.php?do=finduser&u=186377&starteronly=1)? It will display your newly started thread.

Your "alerting solution" is not desirable because it likely would not let me be alerted when other users get responded to and also because I don't want an email alert -- I want a listing of all activity I am interested in. Previously the "search" feature served as a proxy for that.
Let me adopt your type of reply: Lame. The worst thing any customer can tell me is that they know in advance how a new product will work, without understanding the logic behind it or read a detailed product description. How can you assume these things if you did not saw or read anything about the product? DOUBLE FAIL

arn
Nov 28, 2011, 04:21 PM
ok guys. Let's not escalate this further. :)

I'm looking into replacing the feature in another fashion. I'll keep you apprised.

arn

velocityg4
Nov 29, 2011, 10:07 AM
Well I'm quite happy with the new search. Now when I perform an advanced search by user name it displays the results in the date order of the last post a thread has received. Rather than by the last post the username I searched for posted.

I just use this search method to quickly see what threads I have posted in have had any recent activity.

Sometimes a thread may not have activity for months. If it was a technical thread. There could be a question or update that would be good to know.

I long missed this feature and am quite glad it has returned.

b24pgg
Nov 29, 2011, 10:47 AM
If you try to search for posts containing a user name to find replies to a user, its not hitting them all.
Also having this problem

arn
Nov 29, 2011, 11:17 AM
Also having this problem

ya, I think I know the reason for this. looking into it.

arn

arn
Nov 30, 2011, 01:12 PM
As an update for the "who quoted me" feature, we have someone working on it, and if all goes as planned, we should have something ready by next week-ish. Time frames slip of course, so it may be longer.

arn

TECK
Nov 30, 2011, 11:22 PM
I just use this search method to quickly see what threads I have posted in have had any recent activity.

You could also bookmark these links:
All your posts (http://forums.macrumors.com/search.php?do=finduser&u=44688) | All threads started by you (http://forums.macrumors.com/search.php?do=finduser&u=44688&starteronly=1)

The easiest way to see these stats for any user is by visiting his/her profile and click on Statistics tab.

b24pgg
Nov 30, 2011, 11:27 PM
As an update for the "who quoted me" feature, we have someone working on it, and if all goes as planned, we should have something ready by next weak-ish. Time frames slip of course, so it may be longer.

arn
Thanks man

NutsNGum
Dec 2, 2011, 11:25 AM
I highlight text to mark where I'm reading, so this is getting turned off.

I thought I was the only one.

BC2009
Dec 2, 2011, 03:31 PM
As an update for the "who quoted me" feature, we have someone working on it, and if all goes as planned, we should have something ready by next week-ish. Time frames slip of course, so it may be longer.

arn

Thank you!

Any chance we can see who quoted other users? Sometimes you find a great post that may be controversial and you want to quickly look to see how others responded to that person. There are particular posters whom I sometimes follow because its great to see how they respond to those who responded to them.

Thanks again.

GGJstudios
Dec 5, 2011, 06:00 PM
I may have missed where this has already been mentioned, but it appears the search index is updating much slower now. Previously, I could see a new thread in Forum Spy and if I clicked on the poster's profile, then "Find all threads started by" that user, the newly posted thread appeared. Now it doesn't for some time. I'm not sure how long I need to wait before that thread should show up under that user's threads.

TECK
Dec 8, 2011, 10:46 PM
I may have missed where this has already been mentioned, but it appears the search index is updating much slower now. Previously, I could see a new thread in Forum Spy and if I clicked on the poster's profile, then "Find all threads started by" that user, the newly posted thread appeared. Now it doesn't for some time. I'm not sure how long I need to wait before that thread should show up under that user's threads.

The search index is updated every 10 minutes.
I started a new thread 5 minutes ago. Technically, the thread will show under my profile (http://forums.macrumors.com/search.php?do=finduser&u=642422&starteronly=1) in 5 minutes.

GGJstudios
Dec 9, 2011, 04:19 AM
The search index is updated every 10 minutes.
I started a new thread 5 minutes ago. Technically, the thread will show under my profile (http://forums.macrumors.com/search.php?do=finduser&u=642422&starteronly=1) in 5 minutes.
Thanks. It used to be faster than that.

BC2009
Dec 12, 2011, 01:47 PM
Those interested in finding replies to their posts should follow this thread:

http://forums.macrumors.com/showthread.php?t=1283215

and this one (better):

http://forums.macrumors.com/showthread.php?t=1290315