c# - What is the best way to load multiple remote RSS feeds? -

i'm working on project need load multiple (100+) remote rss feeds, parse them , query keywords. process time consuming , i'm looking best way implement this.

my current implementation loads feeds synchronously, because asynchronous implementation tpl failed because there many tasks created during process , throws exception.

the async part loading remote feed looks this:

/// <summary> /// loads specified url. /// </summary> /// <param name="url">the url.</param> /// <returns></returns> /// <exception cref="scanexception">unable download rss feed specified url. check inner exception more details.</exception> protected async task<xdocument> load(string url) {     xdocument document = null;      try     {         using (var client = new httpclient())         {             httpresponsemessage response = await client.getasync(url);              if (response.issuccessstatuscode)             {                 string content = await response.content.readasstringasync();                 document = xdocument.parse(content);             }         }     }     catch (exception ex)     {         throw new scanexception(url, "unable download rss feed specified url. check inner exception more details.", ex);     }      return document; } 

i hope guys can point me in right direction, can work right (performance wise).

the final question is: best way load multiple remote rss feeds?

test code

/// <summary>         /// reads feeds batch async.         /// </summary>         /// <param name="feeds">the feeds.</param>         public void readfeedsbybatchasync(string[] feeds, torrentstorage storage, int batchsize = 8)         {             var tasks = new list<task>(batchsize);             var feedsleft = feeds.length;              foreach (string feed in feeds)             {                 var readfeedtask = this.client.getstringasync(feed);                  if (readfeedtask.status == taskstatus.rantocompletion)                 {                     xdocument document = xdocument.parse(readfeedtask.result);                     var torrents = processxmldocument(document);                      storage.store(torrents);                 }                  tasks.add(readfeedtask);                 --feedsleft;                  if (tasks.count == tasks.capacity || feedsleft == 0)                 {                     var batchtasks = tasks.toarray();                     tasks.clear();                      try                     {                         task.waitall(batchtasks);                     }                     catch (exception)                     {                         throw;                     }                 }             }           } 

i have solved similar issue in fork of gitextensions. dispatching batches of 8 rest api calls creating tasks, , doing task.waitall each batch of 8. bit simplistic, job without complicating code much:

https://github.com/pombeirp/gitextensions/blob/buildserverintegration/plugins/buildserverintegration/teamcityintegration/teamcityadapter.cs#l178.

one thing suggest reusing httpclient class. looks bit of overkill create new instance each request.