07-29 03:32 阅读 158

.NET Core(C#) 使用IHttpClientFactory实现爬虫执行GET和POST请求

原文网址：https://www.cjavapy.com/article/723/

本文主要介绍.NET Core(C#)中，使用HttpClientFactory(IHttpClientFactory)实现简单爬虫，执行GET和POST请求的方法，以及相关的示例代码。

1､.NET Core项目中注册HttpClientFactory

1) ASP .NET Core项目

在Startup.cs中ConfigureService方法中注册HttpClient，代码如下，

services.AddHttpClient();

2) .NET Core项目

.NET Core项目中需要安装引用Microsoft.Extensions.Http和Microsoft.Extensions.Hosting，注册代码如下，

static void Main(string[] args)
   {      var builder = Host.CreateDefaultBuilder(args);

      builder.ConfigureServices(x =>

      {

          x.AddHttpClient();

      });

      Console.ReadKey();

  }

2､配置HTTP请求头Header

1) 注册HttpClient时配置

services.AddHttpClient();
services.AddHttpClient(

或者

   static void Main(string[] args)
        {            var builder = Host.CreateDefaultBuilder(args);
            builder.ConfigureServices(x =>
            {
                x..AddHttpClient("cjavapy", c =>
{
    c.BaseAddress = new Uri("https://api.cjavapy.com/");
    c.DefaultRequestHeaders.Add("content-type","application/json");
    c.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
    c.DefaultRequestHeaders.Add("Referer","");
});
            });
            Console.ReadKey();
        }

2) 执行请求之前配置

var httpClient = context.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
httpClient.BaseAddress = new Uri("https://api.cjavapy.com/");
httpClient.DefaultRequestHeaders.Add("content-type","application/json");
httpClient.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
httpClient.DefaultRequestHeaders.Add("Referer","");

3､使用HttpClientFactory(IHttpClientFactory)执行GET和POST请求

命名空间：

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Reflection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

1) GET请求获取网页代码

static void Main(string[] args)
  {      var builder = Host.CreateDefaultBuilder(args);
      builder.ConfigureServices(x =>
      {
          x.AddHttpClient();
      });      using (var scope = builder.Build().Services.CreateScope())
      {          var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
          httpClient.BaseAddress = new Uri("https://www.baidu.com");
          httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");
          httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");
          Console.WriteLine(httpClient.GetAsync("/").Result.Content.ReadAsStringAsync().Result);
      }
      Console.ReadKey();
  }

2) POST请求提交数据

static void Main(string[] args)
   {       var builder = Host.CreateDefaultBuilder(args);
       builder.ConfigureServices(x =>
       {
           x.AddHttpClient();
       });       using (var scope = builder.Build().Services.CreateScope())
       {           var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
           httpClient.BaseAddress = new Uri("apiUrl");
           httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat");           //httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");           var result="";           try
           {               //2 minute timeout on wait for response
               httpClient.Timeout = new TimeSpan(0, 2, 0);               //Create an HttpRequestMessage object and pass it into SendAsync()
               HttpRequestMessage message = new HttpRequestMessage();
               message.Headers.Add("Accept", "application/json");
               message.Content = new StringContent("{\"user\":\"11\"}", System.Text.Encoding.UTF8, "application/json");
               message.Method = HttpMethod.Post;
               message.RequestUri = new Uri(httpClient.BaseAddress.ToString() + "someapiendpoint");
               HttpResponseMessage response = httpClient.SendAsync(message).Result;
               result = response.Content.ReadAsStringAsync().Result;               //deserialize the result into proper object type
           }           catch (Exception ex)
           {               //Log exception
           }
       }
       Console.ReadKey();
   }

来源https://www.cnblogs.com/zxtceq/articles/15071644.html