java - Best way to quickly and repeatedly search if value is present in a rapidly growing array? -

i have program looks hundreds of categories website , of each category grabs data product detail pages of many of products in each category

each category them select table 800 categories, , 100 products each category.

the problem many of products belong more 1 category trying put way in code not go product detail page if have grabbed product before(in different category)

so code conceptually this:

thesql = "select catid categories";  resultset rs = statement.executequery(thesql);  while (rs.next()) {   > go check website particular catid    > loop check products in page           > each productid listed in category's page:        > check array see if have encountered productid before(in          session)           > if have, skip product , continue next 1           > if haven't, go product's detail web page, grab             data, insert in database, , add productid in our array.  } 

i have 2 questions:

1) counterproductive or resource intensive have add 1 product id such big array each pass , have search each time see if current product id preexisting in array?(the array potentially end 2000-6000 items)

2) if there more 1 way go it, way recommend?

(please note have litle experience java arrays, though understand concept)

i know use "insert ignore into..." when inserting database table insure not insert duplicates, want save time , resources needed check websites of products have checked.

many thanks!

edit/update: forgot mention productid's not numbers strings of 10 characters, mixing letters , numbers. not sure if makes big difference.

simply use hashset<id> instead of array:

this class offers constant time performance basic operations (add, remove, contains , size), assuming hash function disperses elements among buckets.

10k items won't problem @ all if id implements not intensive hashcode/equals, instance if ids long.

also keep in mind if access web pages , perform db calls, amount of time spent in java code gonna negligible: of time gonna spent waiting external calls return.