# How to implement find\_next() so that a large list could be traversed quickly?

**URL:** <https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473>\
**Category:** Other Northbound Interfaces\
**Created:** [May 20, 2019, 8:36am UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473 "2019-05-20T08:36:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sarahwa](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sarahwa](https://dmap-community.ductus.global/u/sarahwa)\
**Post date:** [May 20, 2019, 8:36am UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/1 "2019-05-20T08:36:51Z")

</div>

Below Yang module is designed for operational data.

```
container files {
                    config false;
                    list file {
                        key file-number;
                        leaf file-number {
                            description "Number of the file";
                            type uint32;
                        }
                        container file-information {
			    description "Information about the file";
                            leaf file-state {
                                description "State of the file";
                                type base:file-state-read;
                            }
                            leaf filling-time {
                                description "The filling timestamp showing when the file is opened";
                                type yang:date-and-time;
			    }
                        leaf transfer-time {
                               description "the transfer date and time"
                               type yang:date-and-time;
                        }
                        leaf is-stored-file {
                            description "Set if the file is stored on disk.";
                            type empty;
                        }
			    leaf is-compressed-file {
                                description "Set if the file is compressed.";
                                type empty;
                            }
                        leaf file-warning {
                                description "Mutually exclusive warnings set for file
                                type enumeration {
                                enum skipped
                                enum overwritten
                                }
                         }                
			}
		    }
 }

```

We would like to use confd\_cli command **“show files file xxx file-information file-state”** to display the file xxx’s status. The file list has a maximum of 65535 elements and find\_next() is design to traverse the list. However, it takes much time to traverse all the elements in the list. （Note: the files are always on a fixed position like this: the file-number=1 is on the position 0; the file-number=2 is on the position 1; the file-number=N is on the position N-1)

```
int find_next_nested (struct confd_trans_ctx* tctx,
                                     confd_hkeypath_t* keypath,
                                     enum confd_find_next_type /*type*/,
                                     confd_value_t* keys,
                                     int nkeys)
{
    confd_value_t v;
    int pos = -1;
    std::vector<FileInfo> current_file_info = findFileInfoList(keypath->v[2][0]);
    switch (nkeys) {
    case 0:
        selected_fileInfo.file_number = UINT_MAX;
        if (current_file_info.size() > 0)
            pos = 0;
        else
            pos = -1;
        break;
    case 1:
        pos = CONFD_GET_UINT32(&keys[0]);
        if (static_cast<int>(current_file_info.size())<=pos)
            pos = -1;
        break;
    default:
        confd_trans_seterr(tctx, "invalid number of keys: %d", nkeys);
        return CONFD_ERR;
    }

    if (pos >= 0) {
        CONFD_SET_UINT32(&v, current_file_info[pos].file_number);
        confd_data_reply_next_key(tctx, &v, 1, (long)(pos + 1));
    } else {
        confd_data_reply_next_key(tctx, NULL, -1, -1);
    }

    return CONFD_OK;
}

```

I tried to change find\_next() to make a “jump” in a list traversal, it seems the result doesn’t satisfy our needs.

```
xxx@netconf-server- 02:52:56> show files file 1
Possible completions:
  100 - Number (running counter) of the file
  1000 - Number (running counter) of the file
  1100 - Number (running counter) of the file
  1200 - Number (running counter) of the file
  1300 - Number (running counter) of the file
  1400 - Number (running counter) of the file
  1500 - Number (running counter) of the file
  1600 - Number (running counter) of the file
  1700 - Number (running counter) of the file
  1800 - Number (running counter) of the file

```

Could you please give some suggestion how to change find\_next() to show any file’s status quickly? Or could I use “tailf:display-when” to forbidden the input of " file-information file-state" in confd-cli? We are using ConfD 6.2.

---

<div class="post-metadata">

**Author:** ![mvf](https://yyz2.discourse-cdn.com/flex010/user_avatar/dmap-community.ductus.global/mvf/32/182_2.png) [@mvf](https://dmap-community.ductus.global/u/mvf)\
**Post date:** [May 22, 2019, 1:36pm UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/2 "2019-05-22T13:36:36Z")

</div>

I don’t think you can do much more in `find_next` to improve the performance. Have a look at developer log and at your application log - I expect you will see lots of `get_next` invocations, likely this is what is taking time, and you can’t improve that by changing your `find_next`.

---

<div class="post-metadata">

**Author:** ![sarahwa](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sarahwa](https://dmap-community.ductus.global/u/sarahwa)\
**Post date:** [May 23, 2019, 6:37am UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/3 "2019-05-23T06:37:14Z")

</div>

Hi, mvf,

Yes, I can see lots of find\_next() invoked in application log.

I am trying to implement get\_next\_object to retrieve the entire list in one invoke. However, I am encountering the error:

```
**"TRACE CALL data get_next_object(thandle=25, /files/file, -1)DEBUG Invalid confd_vtype value: 33"**
 **"DEBUG Expect call to return data, cannot return CONFD_OK when no data is sent thandle=25"**
**"Error on worker socket request: Bad protocol usage or unexpected retval (21): Expect call to return data, cannot return CONFD_OK when no data is sent thandle=25"**

```

Could you help point out which is wrong in my implementation?

```auto
    int get_next_object (struct confd_trans_ctx *tctx,
                                             confd_hkeypath_t *keypath,
                                             long next)
    {

      int pos = -1;

      confd_value_t *v;
      struct confd_next_object *obj;
      int i;

      int n_list_entries;
      FileInfo file_info;
      std::vector<FileInfo> curr_file_list = tctx->t_opaque;
      
      n_list_entries = curr_file_list.size();

      if (next == -1)
      {
        if(n_list_entries > 0) {
          pos = 0; 
        }
      }else if (next < n_list_entries) {
        pos = next;
      } else {
        confd_data_reply_next_object_array(tctx, NULL, -1, -1);
        return CONFD_OK;
      }

    v = (confd_value_t *) malloc(sizeof(confd_value_t) * n_list_entries * 8);
      obj = (struct confd_next_object *)malloc(sizeof(struct confd_next_object) * (n_list_entries + 1));

      if (pos != -1) {
        for (i = 0; pos + i < n_list_entries; i++) {
          file_info = curr_file_list[pos + i];
          obj[i].v = &v[i * 8];

          CONFD_SET_UINT32(&(obj[i].v[0]), file_info.file_number);
          CONFD_SET_XMLTAG(&(obj[i].v[1]), streams_file_information, streams__ns);
          CONFD_SET_ENUM_VALUE(&(obj[i].v[2]), get_enum_file_state(file_info.file_state));
          CONFD_SET_DATETIME(&(obj[i].v[3]), get_date_time(file_info.filling_time));
          CONFD_SET_DATETIME(&(obj[i].v[4]), get_date_time(file_info.transfer_time));
          CONFD_SET_BOOL(&(obj[i].v[5]), file_info.is_stored_file);
          CONFD_SET_BOOL(&(obj[i].v[6]), file_info.is_compressed_file);
          CONFD_SET_ENUM_VALUE(&(obj[i].v[7]), file_info.file_warning);
          obj[i].n = 8;
          obj[i].next = -1;
        }
        if (pos + i == n_list_entries)
          obj[i++].v = NULL;
        confd_data_reply_next_object_arrays(tctx, obj, i, 0);
      } else {
        confd_data_reply_next_object_array(tctx, NULL, -1, -1);
      }

      free(v);
      free(obj);
      return CONFD_OK;
    }

```

Thanks in advance,  
sarahwa

---

<div class="post-metadata">

**Author:** ![mvf](https://yyz2.discourse-cdn.com/flex010/user_avatar/dmap-community.ductus.global/mvf/32/182_2.png) [@mvf](https://dmap-community.ductus.global/u/mvf)\
**Post date:** [May 23, 2019, 10:46am UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/4 "2019-05-23T10:46:25Z")

</div>

I cannot tell much from this, but your code obviously does not correspond to the data model you pasted in the first post: there are no `transfer-time` and `file-warning` leaves in the data model, and the leaf `is-compressed-file` is empty, but you fill in boolean. It does not look like this is what causes libconfd to complain though. I suggest you debug your code to find why the vtype 33 (this is `C_XMLEND`) appeared there.

---

<div class="post-metadata">

**Author:** ![sarahwa](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sarahwa](https://dmap-community.ductus.global/u/sarahwa)\
**Post date:** [May 23, 2019, 12:11pm UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/5 "2019-05-23T12:11:49Z")

</div>

Hi mvf,  
Thanks for your info. I have updated the yang module in first post.  
Besides, I have implemented exists\_optional() for is-stored-file and is-compressed-file.  
I am a newbie to confd. Could you tell me how to debug the code? Is it fine to set the debug level to CONFD\_PROTO\_TRACE?

Thanks,  
sarahwa

---

<div class="post-metadata">

**Author:** ![mvf](https://yyz2.discourse-cdn.com/flex010/user_avatar/dmap-community.ductus.global/mvf/32/182_2.png) [@mvf](https://dmap-community.ductus.global/u/mvf)\
**Post date:** [May 23, 2019, 12:42pm UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/6 "2019-05-23T12:42:26Z")

</div>

`CONFD_PROTO_TRACE` would not be very useful to you. What I meant is a standard debugging process - adding logs, using a debugger, whatever works for you; the fact that your application talks to confd makes no difference here.

I still believe you should not use `CONFD_SET_BOOL` for an empy-typed leaf in a response to `get_next_object` - correct usage would be `CONFD_SET_XMLTAG` if the leaf exists, `CONFD_SET_NOEXISTS` otherwise.

---

<div class="post-metadata">

**Author:** ![sarahwa](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sarahwa](https://dmap-community.ductus.global/u/sarahwa)\
**Post date:** [May 24, 2019, 6:05am UTC](https://dmap-community.ductus.global/t/how-to-implement-find-next-so-that-a-large-list-could-be-traversed-quickly/2473/7 "2019-05-24T06:05:54Z")

</div>

Hi mvf,

I change the get\_next\_object() for empty-typed leaf as below and it works well.

```
  if (file_info.is_stored_file)
  {
    CONFD_SET_XMLTAG(&(obj[i].v[5]), streams_is_stored_file, streams__ns);
  } else {
    CONFD_SET_NOEXISTS(&(obj[i].v[5]));
  }

  if (file_info.is_compressed_file)
  {
    CONFD_SET_XMLTAG(&(obj[i].v[6]), streams_is_compressed_file, streams__ns);
  } else {
    CONFD_SET_NOEXISTS(&(obj[i].v[6]));
  }

```

Thanks very much,  
sarahwa